Native Django on Google App Engine – A JSON RESTful Web Service (Sample Project)
This post is for anyone interested in creating a RESTful Web Service API with Django running on Google App Engine.
I've attached the project code, you can clone it.
I developed the project using Aptana Studio 3 with Google plugins, but you should be able to import it in Eclipse, if you've installed the right tools (PyDev and, again, Google plugins).
I used JSON, and It's running native Django on Google App Engine, using Django-nonrel.
Native Django on App Engine
I'll give you a brief description on how to run native Django on App Engine, this means, the ability to use the standard django models instead of the one provided by App Engine, without losing most of the Django functionalities and existing applications. The solution is based on Django-nonrel, it's as simple as clone some repositories (django-nonrel, djangoappengine, and djangotoolbox) including them into an app engine project, and tweaking some configuration files. Luckily, there's also a sample app that includes the configuration files and all the things you need to get started with a native Django project running on App Engine.
You can find the step by step guide here.
I've included everything in the source code, I won't cover the Native Django integration, cause the link I provided describes it well. So, if you want to try creating the base project by yourself, follow the link instructions.
Directory Structure
I modified the directory structure a little, because I like having things organized in my way, the directory structure of my project is the following:
lib: a folder containing 3rd party apps and the django-nonrel, djangoappengine, and django-toolbox helpers
media: contains the media static files (css, js, images, etc..)
project: the project folder, where I put the main modules (manage.py, settings.py, urls.py, views.py)
templates: the django templates folder
The other folders under the root are the django applications of the project, in this case the applications composing the project are: rest, registration, and messaging
If you modify the directory structure you'll need to fix the configuration files. Modify app.yaml, if you change path or the content of the lib folder, and modify the djangoappengine/boot.py file if you change the path of Django settings.py file.
Project Description
The project has 2 little REST API, one for user registration, and another for sending messages from one user to another, fetching the messages, and deleting them (for staff users).
Note I know it's not great to have the registration of the users handled by a web service call, this is just a demonstrative example, registration should be better off done by filling in a form.
The functionalities of this projects are the following:
It uses a RESTView, which you can subclass to create the views of your API, and dispatches the request based on HTTP method
It has 2 decorators, one for handling basic authentication, and another for throttling requests
Some little utility functions to handle JSON Requests and Responses, plus a class StatusCodes that contains the main Http Response Status Codes you'll use in a RESTful API
The REST View
The first part of the REST application is the RESTView:
class RESTView(object):
"""
Dispatches a request based on HTTP method
"""
methods = ['GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS']
def __call__(self, request, *args, **kwargs):
callback = getattr(self, request.method, None)
if callback:
return callback(request, *args, **kwargs)
else:
allowed_methods = self.get_allowed_methods()
return http.HttpResponseNotAllowed(allowed_methods)
def get_allowed_methods(self):
return [m for m in self.methods if hasattr(self, m) and m != 'OPTIONS']
def OPTIONS(self, request):
allowed_methods = self.get_allowed_methods()
response = http.HttpResponse(status=StatusCodes.OK)
response['Allow'] = ', '.join(allowed_methods)
return responseBy subclassing the view you can write methods for each specific http method the api should respond to.
Inside the variable methods I've put the basic HTTP Methods that a REST API should have, when a Request is received at the url of a specific REST View, it will checkif the Method is part of the methods list and if it has been implemented in the class, if it's not, it will return a Method Not Allowed Response, specifying the allowed methods in the Allow header of the response.
The HTTP method OPTIONS exists to let a client know which methods are allowed by a specific url. You won't need to write this method in your subclasses, it is handled automatically by the base view, and it returns the Allow header, specifying the methods you have implemented in your REST views.
HTTP Methods
If you need a little description of the main HTTP methods, and their expected behavior for a RESTful API, read this little list, if you already know everything about REST, feel free to jump to the Registration View.
GET: Retrieve a representation of a resource
POST: Create a new resource to an existing uri
PUT: Create a new resource to a new uri/Modify a resource to an existing uri
DELETE: Delete an existing resource
HEAD: Retrieve metadata-only representation
OPTIONS: Check which HTTP methods a particular resource supports
The Registration REST View
This is the first example of subclass of the REST view, I'm showing you the registration application, it consists of just a view, handling POST requests, and user registration.
class RegistrationApi(RESTView):
@throttle(1,1*60,"UserRegistration")
def POST(self, request):
"""
Accepts a JSON request containing:
- username
- email
- password
and saves the User on the backend
"""
try:
data = JSONObject(request)
except:
return http.HttpResponseBadRequest()
username = force_unicode(data.get('username',''))
email = force_unicode(data.get('email',''))
password = force_unicode(data.get('password',''))
if username == '' or email == '' or password == '':
return http.HttpResponseBadRequest()
try:
User.objects.get(username=username)
return http.HttpResponse(status=StatusCodes.DUPLICATE_ENTRY)
except User.DoesNotExist:
User.objects.create_user(username,email,password)
return http.HttpResponse(status=StatusCodes.CREATED)Ignore the throttle decorator for now, I'll cover it later on. I'm not showing you the url file, but it just map this view to the url api/registration/.
The class is a subclass of RESTView, and it defines the method POST, you can imagine that this will handle POST requests, easy enough.
Example content of a request is the following:
{ "username": "Bruno", "email": "bruno@bfil.io", "password": "my_secret_password" }The first thing it does is passing the request to the JSONObject utility function, it parses the content of a request (in JSON format) into a dictionary object and returns the object.
def JSONObject(request):
"""
Returns a JSON object based on the content of an HttpRequest
"""
return simplejson.loads(request.raw_post_data)It the content can't be parsed the server will return a Bad Request Response. I haven't added much validation so you won't lose the focus on what's important in developing a RESTful API. For example in this case email validation should be added.
It checks if an user with the same username already exists before creating it, if it already exists, it returns a Response with Status Conflict/Duplicate Entry. StatusCodes is just a class containing the main HTTP Response Codes:
class StatusCodes(object):
"""
Provides a verbose list of Status Codes for HttpResponse objects
"""
OK = 200
CREATED = 201
DELETED = 204
BAD_REQUEST = 400
FORBIDDEN = 401
NOT_FOUND = 404
DUPLICATE_ENTRY = 409
NOT_HERE = 410
INTERNAL_ERROR = 500
NOT_IMPLEMENTED = 501
THROTTLED = 503Basic Authentication and Throttling Decorators
Inside the rest applications you'll find 2 decorators, one for handling Basic Authentication for requests, and another one for throttling requests.
I've been inspired by some django snippets for the Basic Authentication decorator, and I've taken the Throttling decorator from django-piston.
The code of basicauth is the following:
def basicauth(realm = ""):
"""
Decorator for views that checks the basic authentication header
"""
def view_decorator(func):
def wrapper(self,request, *args, **kwargs):
if 'HTTP_AUTHORIZATION' in request.META:
auth = request.META['HTTP_AUTHORIZATION'].split()
if len(auth) == 2:
if auth[0].lower() == "basic":
username, password = base64.b64decode(auth[1]).split(':')
user = authenticate(username=username, password=password)
if user:
request.user = user
return func(self, request, *args, **kwargs)
response = http.HttpResponse()
response.status_code = StatusCodes.FORBIDDEN
response['WWW-Authenticate'] = 'Basic realm="%s"' % realm
return response
return wrapper
return view_decoratorIt just checks the presence of the HTTP_AUTHORIZATION header, and if it is present it checks the user authentication. If authentication fails, or the Authorization header isn't provided, the response will have status Forbidden, and it will attach the WWW-Authenticate header to it, to ask for basic authentication.
If everything is OK it attaches the user to the request object, so that you can do other stuff inside the view itself.
You can specify a Realm as a parameter if you want.
The Throttling decorator:
def throttle(max_requests, timeout=60*60, extra=''):
"""
Decorator for views that throttles the requests from an user or ip address
that exceeds the maximum number of requests in a given time
"""
def view_decorator(func):
def wrapper(self, request, *args, **kwargs):
user = request.user;
if user:
ident = user.username
else:
ident = request.META.get('REMOTE_ADDR', None)
if ident:
ident += ':%s' % extra
now = time.time()
count, expiration = cache.get(ident, (1, None))
if expiration is None:
expiration = now + timeout
if count >= max_requests and expiration > now:
response = http.HttpResponse(status=StatusCodes.THROTTLED)
wait = int(expiration - now)
response.content = 'Throttled, wait %d seconds.' % wait
response['Retry-After'] = wait
return response
cache.set(ident, (count+1, expiration), (expiration - now))
return func(self, request, *args, **kwargs)
return wrapper
return view_decoratorIt has 3 parameters:
max_requests: sets the maximum number of request to handle before throttling them
timeout (in seconds): it will reset the request count each time it is elapsed
extra (a string): to group/divide request counts
The Messaging Application
Inside the project there's another application, I called it messaging.
It provides a model and a rest view.
The model is the following:
class Message(models.Model):
"""
An object containing a message sent from an User to another
"""
from_user = models.ForeignKey(User, related_name="from_user")
to_user = models.ForeignKey(User, related_name="to_user")
message = models.TextField()
datetime = models.DateTimeField()
def __unicode__(self):
return self.message
def format(self):
"""
Returns a dictionary rapresenting the object in JSON format
It can be used to create a JSONResponse
"""
info = {
'from_user': self.from_user.username,
'to_user': self.to_user.username,
'message': self.message,
'datetime': force_unicode(self.datetime)
}
return info
def parse(self, request):
"""
Parses the content of an HttpRequest to a Message object
"""
try:
data = JSONObject(request)
self.from_user = request.user
self.to_user = User.objects.get(username=force_unicode(data.get('to_user','')))
self.message = force_unicode(data.get('message', ''))
self.datetime = datetime.datetime.now()
except (ValueError, KeyError, TypeError):
passYou can see that the model is a subclass of the native Django models, the helpers we included into the project handles the App Engine backend, so we don't have to worry about anything, except that we can't use JOINs for our queries, it's the well-known limit of non-relational databases.
The model has 2 methods for handling serialization/deserialization into JSON format. There are better ways to do this, you just need a good JSON serializer that handles this stuff automatically, or you can write your own.
I kept this simple, the format() method returns a dictionary that can become easily a JSON string, the parse() method parses a request into a Message object.
The rest api for the messaging application provides three methods: GET, POST and DELETE, and it is mapped to the url: api/messaging/
class MessagingApi(RESTView):
@basicauth(realm=REALM)
@throttle(60,1*60,"GetMessages")
def GET(self, request):
"""
Returns the last 50 messages sent/received by the user of the request in JSON format
The messages can be filtered specifying the parameters in the url:
- to_user: the username of another user to get a specific conversation
- datetime: the minimum datetime for the messages returned
"""
sent = Message.objects.all().filter(from_user = request.user)
received = Message.objects.all().filter(to_user = request.user)
if request.GET.has_key('to_user'):
to_user = None
try:
to_user = User.objects.get(username = request.GET['to_user'])
except User.DoesNotExist:
return http.HttpResponse(status=StatusCodes.NOT_FOUND)
if to_user:
sent = sent.filter(to_user = to_user)
received = received.filter(from_user = to_user)
if request.GET.has_key('datetime'):
datetime = parse(smart_str(request.GET['datetime']))
sent = sent.filter(datetime__gt = datetime)
received = received.filter(datetime__gt = datetime)
sent = sent.order_by("-datetime")[:50]
received = received.order_by("-datetime")[:50]
messages = [item for item in sent]
for item in received:
messages.append(item)
sorted_messages = sorted(messages, key=lambda m: m.datetime)
return JSONResponse([item.format() for item in sorted_messages])
@basicauth(realm=REALM)
@throttle(10,1*60,"PostMessage")
def POST(self, request):
"""
Parses the JSON request to create a Message object
"""
item = Message()
item.parse(request)
if not item:
return http.HttpResponseBadRequest()
item.save()
return http.HttpResponse(status=StatusCodes.CREATED)
@basicauth(realm=REALM)
def DELETE(self, request):
"""
Deletes all the messages on the database (must be a staff user)
"""
if request.user.is_staff:
Message.objects.all().delete()
return http.HttpResponse(status=StatusCodes.DELETED)
else:
response = http.HttpResponseForbidden()
return responseAll the methods requires basic authentication, so you'll need to register an user with the registration api, so then you can then provide the authentication for the messaging api.
The GET method accepts GET parameters in the URL and returns the latest 50 messages related to the user making the request.
You can specify the to_user parameter, if you want to fetch the messages of a specific conversation, and a datetime parameter, if you want to have a minimum date settled.
Example of a GET request:
api/messaging/?to_user=Bruno&datetime=2010-12-12 04:00:00I created an utility class, JSONResponse, subclassing HttpResponse
class JSONResponse(http.HttpResponse):
"""
Creates an instance of HttpResponse containing JSON content
"""
def __init__(self, data):
indent = 2 if settings.DEBUG else None
mime = ("text/javascript" if settings.DEBUG
else "application/json")
super(JSONResponse, self).__init__(
content = simplejson.dumps(data, indent=indent),
mimetype = mime,
)It's used to create a json response based on python dictionary objects, and lists of dictionary objects.
In MessagingApi GET case it passes a list of formatted Messages to JSONResponse, that outputs them in the content of the response in JSON format.
The POST method accepts a JSON input rapresenting a message, it parse the request to a Message object model, and saves it to the datastore. If it can't parse the request it returns Bad Request status, if everything is ok, it returns the status Created
Example of a POST Request:
{ "to_user": "God", "message": "Hi God!" }The DELETE method checks if the user is part of the staff (you can modify an user by going into the Datastore Viewew in the SDK Console, or typing the url: _ah/admin/datastore/), if it is, it deletes all the messages from the datastore, and returns status Deleted.
I hope you'll find this useful for your REST knowledge.
Update (08/02/2011)
I updated the source to include the latest release of django-nonrel, djangoappengine and django-toolbox