Native Django on Google App Engine – A JSON RESTful Web Service (Sample Project)

This post is for anyone interested in creating a RESTful Web Service API with Django running on Google App Engine.

I've attached the project code, you can clone it.

I developed the project using Aptana Studio 3 with Google plugins, but you should be able to import it in Eclipse, if you've installed the right tools (PyDev and, again, Google plugins).

I used JSON, and It's running native Django on Google App Engine, using Django-nonrel.

Native Django on App Engine

I'll give you a brief description on how to run native Django on App Engine, this means, the ability to use the standard django models instead of the one provided by App Engine, without losing most of the Django functionalities and existing applications. The solution is based on Django-nonrel, it's as simple as clone some repositories (django-nonrel, djangoappengine, and djangotoolbox) including them into an app engine project, and tweaking some configuration files. Luckily, there's also a sample app that includes the configuration files and all the things you need to get started with a native Django project running on App Engine.

You can find the step by step guide here.

I've included everything in the source code, I won't cover the Native Django integration, cause the link I provided describes it well. So, if you want to try creating the base project by yourself, follow the link instructions.

Directory Structure

I modified the directory structure a little, because I like having things organized in my way, the directory structure of my project is the following:

  • lib: a folder containing 3rd party apps and the django-nonrel, djangoappengine, and django-toolbox helpers

  • media: contains the media static files (css, js, images, etc..)

  • project: the project folder, where I put the main modules (manage.py, settings.py, urls.py, views.py)

  • templates: the django templates folder

The other folders under the root are the django applications of the project, in this case the applications composing the project are: rest, registration, and messaging

If you modify the directory structure you'll need to fix the configuration files. Modify app.yaml, if you change path or the content of the lib folder, and modify the djangoappengine/boot.py file if you change the path of Django settings.py file.

Project Description

The project has 2 little REST API, one for user registration, and another for sending messages from one user to another, fetching the messages, and deleting them (for staff users).

Note I know it's not great to have the registration of the users handled by a web service call, this is just a demonstrative example, registration should be better off done by filling in a form.

The functionalities of this projects are the following:

  • It uses a RESTView, which you can subclass to create the views of your API, and dispatches the request based on HTTP method

  • It has 2 decorators, one for handling basic authentication, and another for throttling requests

  • Some little utility functions to handle JSON Requests and Responses, plus a class StatusCodes that contains the main Http Response Status Codes you'll use in a RESTful API

The REST View

The first part of the REST application is the RESTView:

class RESTView(object):
    """
    Dispatches a request based on HTTP method
    """
    methods = ['GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS']

    def __call__(self, request, *args, **kwargs):
        callback = getattr(self, request.method, None)
        if callback:
            return callback(request, *args, **kwargs)
        else:
            allowed_methods = self.get_allowed_methods()
            return http.HttpResponseNotAllowed(allowed_methods)

    def get_allowed_methods(self):
        return [m for m in self.methods if hasattr(self, m) and m != 'OPTIONS']

    def OPTIONS(self, request):
        allowed_methods = self.get_allowed_methods()
        response = http.HttpResponse(status=StatusCodes.OK)
        response['Allow'] = ', '.join(allowed_methods)
        return response

By subclassing the view you can write methods for each specific http method the api should respond to.

Inside the variable methods I've put the basic HTTP Methods that a REST API should have, when a Request is received at the url of a specific REST View, it will checkif the Method is part of the methods list and if it has been implemented in the class, if it's not, it will return a Method Not Allowed Response, specifying the allowed methods in the Allow header of the response.

The HTTP method OPTIONS exists to let a client know which methods are allowed by a specific url. You won't need to write this method in your subclasses, it is handled automatically by the base view, and it returns the Allow header, specifying the methods you have implemented in your REST views.

HTTP Methods

If you need a little description of the main HTTP methods, and their expected behavior for a RESTful API, read this little list, if you already know everything about REST, feel free to jump to the Registration View.

  • GET: Retrieve a representation of a resource

  • POST: Create a new resource to an existing uri

  • PUT: Create a new resource to a new uri/Modify a resource to an existing uri

  • DELETE: Delete an existing resource

  • HEAD: Retrieve metadata-only representation

  • OPTIONS: Check which HTTP methods a particular resource supports

The Registration REST View

This is the first example of subclass of the REST view, I'm showing you the registration application, it consists of just a view, handling POST requests, and user registration.

class RegistrationApi(RESTView):

    @throttle(1,1*60,"UserRegistration")
    def POST(self, request):
        """
        Accepts a JSON request containing:
            - username
            - email
            - password
        and saves the User on the backend
        """
        try:
            data = JSONObject(request)
        except:
            return http.HttpResponseBadRequest()

        username = force_unicode(data.get('username',''))
        email = force_unicode(data.get('email',''))
        password = force_unicode(data.get('password',''))

        if username == '' or email == '' or password == '':
            return http.HttpResponseBadRequest()

        try:
            User.objects.get(username=username)
            return http.HttpResponse(status=StatusCodes.DUPLICATE_ENTRY)
        except User.DoesNotExist:
            User.objects.create_user(username,email,password)
            return http.HttpResponse(status=StatusCodes.CREATED)

Ignore the throttle decorator for now, I'll cover it later on. I'm not showing you the url file, but it just map this view to the url api/registration/.

The class is a subclass of RESTView, and it defines the method POST, you can imagine that this will handle POST requests, easy enough.

Example content of a request is the following:

{ "username": "Bruno", "email": "bruno@bfil.io", "password": "my_secret_password" }

The first thing it does is passing the request to the JSONObject utility function, it parses the content of a request (in JSON format) into a dictionary object and returns the object.

def JSONObject(request):
    """
    Returns a JSON object based on the content of an HttpRequest
    """
    return simplejson.loads(request.raw_post_data)

It the content can't be parsed the server will return a Bad Request Response. I haven't added much validation so you won't lose the focus on what's important in developing a RESTful API. For example in this case email validation should be added.

It checks if an user with the same username already exists before creating it, if it already exists, it returns a Response with Status Conflict/Duplicate Entry. StatusCodes is just a class containing the main HTTP Response Codes:

class StatusCodes(object):
    """
    Provides a verbose list of Status Codes for HttpResponse objects
    """ 
    OK = 200
    CREATED = 201
    DELETED = 204
    BAD_REQUEST = 400
    FORBIDDEN = 401
    NOT_FOUND = 404
    DUPLICATE_ENTRY = 409
    NOT_HERE = 410
    INTERNAL_ERROR = 500
    NOT_IMPLEMENTED = 501
    THROTTLED = 503

Basic Authentication and Throttling Decorators

Inside the rest applications you'll find 2 decorators, one for handling Basic Authentication for requests, and another one for throttling requests.

I've been inspired by some django snippets for the Basic Authentication decorator, and I've taken the Throttling decorator from django-piston.

The code of basicauth is the following:

def basicauth(realm = ""):
    """
    Decorator for views that checks the basic authentication header
    """
    def view_decorator(func):
        def wrapper(self,request, *args, **kwargs):
            if 'HTTP_AUTHORIZATION' in request.META:
                auth = request.META['HTTP_AUTHORIZATION'].split()
                if len(auth) == 2:
                    if auth[0].lower() == "basic":
                        username, password = base64.b64decode(auth[1]).split(':')

                        user = authenticate(username=username, password=password)

                        if user:
                            request.user = user
                            return func(self, request, *args, **kwargs)

            response = http.HttpResponse()
            response.status_code = StatusCodes.FORBIDDEN
            response['WWW-Authenticate'] = 'Basic realm="%s"' % realm
            return response
        return wrapper
    return view_decorator

It just checks the presence of the HTTP_AUTHORIZATION header, and if it is present it checks the user authentication. If authentication fails, or the Authorization header isn't provided, the response will have status Forbidden, and it will attach the WWW-Authenticate header to it, to ask for basic authentication.

If everything is OK it attaches the user to the request object, so that you can do other stuff inside the view itself.

You can specify a Realm as a parameter if you want.

The Throttling decorator:

def throttle(max_requests, timeout=60*60, extra=''):
    """
    Decorator for views that throttles the requests from an user or ip address
    that exceeds the maximum number of requests in a given time
    """
    def view_decorator(func):
        def wrapper(self, request, *args, **kwargs):
            user = request.user;
            if user:
                ident = user.username
            else:
                ident = request.META.get('REMOTE_ADDR', None)

            if ident:
                ident += ':%s' % extra

                now = time.time()
                count, expiration = cache.get(ident, (1, None))

                if expiration is None:
                    expiration = now + timeout

                if count >= max_requests and expiration > now:
                    response = http.HttpResponse(status=StatusCodes.THROTTLED)
                    wait = int(expiration - now)
                    response.content = 'Throttled, wait %d seconds.' % wait
                    response['Retry-After'] = wait
                    return response

                cache.set(ident, (count+1, expiration), (expiration - now))

            return func(self, request, *args, **kwargs)
        return wrapper
    return view_decorator

It has 3 parameters:

  • max_requests: sets the maximum number of request to handle before throttling them

  • timeout (in seconds): it will reset the request count each time it is elapsed

  • extra (a string): to group/divide request counts

The Messaging Application

Inside the project there's another application, I called it messaging.

It provides a model and a rest view.

The model is the following:

class Message(models.Model):
    """
    An object containing a message sent from an User to another
    """
    from_user = models.ForeignKey(User, related_name="from_user")
    to_user = models.ForeignKey(User, related_name="to_user")
    message = models.TextField()
    datetime = models.DateTimeField()

    def __unicode__(self):
        return self.message

    def format(self):
        """
        Returns a dictionary rapresenting the object in JSON format
        It can be used to create a JSONResponse
        """
        info = {
            'from_user': self.from_user.username,
            'to_user': self.to_user.username,
            'message': self.message,
            'datetime': force_unicode(self.datetime)
        }
        return info

    def parse(self, request):
        """
        Parses the content of an HttpRequest to a Message object
        """
        try:
            data = JSONObject(request)
            self.from_user = request.user
            self.to_user = User.objects.get(username=force_unicode(data.get('to_user','')))
            self.message = force_unicode(data.get('message', ''))
            self.datetime = datetime.datetime.now()
        except (ValueError, KeyError, TypeError):
            pass

You can see that the model is a subclass of the native Django models, the helpers we included into the project handles the App Engine backend, so we don't have to worry about anything, except that we can't use JOINs for our queries, it's the well-known limit of non-relational databases.

The model has 2 methods for handling serialization/deserialization into JSON format. There are better ways to do this, you just need a good JSON serializer that handles this stuff automatically, or you can write your own.

I kept this simple, the format() method returns a dictionary that can become easily a JSON string, the parse() method parses a request into a Message object.

The rest api for the messaging application provides three methods: GET, POST and DELETE, and it is mapped to the url: api/messaging/

class MessagingApi(RESTView):

    @basicauth(realm=REALM)
    @throttle(60,1*60,"GetMessages")
    def GET(self, request):
        """
        Returns the last 50 messages sent/received by the user of the request in JSON format

        The messages can be filtered specifying the parameters in the url:
            - to_user: the username of another user to get a specific conversation
            - datetime: the minimum datetime for the messages returned
        """
        sent = Message.objects.all().filter(from_user = request.user)
        received = Message.objects.all().filter(to_user = request.user)

        if request.GET.has_key('to_user'):

            to_user = None

            try:
                to_user = User.objects.get(username = request.GET['to_user'])
            except User.DoesNotExist:
                return http.HttpResponse(status=StatusCodes.NOT_FOUND)

            if to_user:
                sent = sent.filter(to_user = to_user)
                received = received.filter(from_user = to_user)

        if request.GET.has_key('datetime'):
            datetime = parse(smart_str(request.GET['datetime']))
            sent = sent.filter(datetime__gt =  datetime)
            received = received.filter(datetime__gt = datetime)

        sent = sent.order_by("-datetime")[:50]
        received = received.order_by("-datetime")[:50]

        messages = [item for item in sent]
        for item in received:
            messages.append(item)

        sorted_messages = sorted(messages, key=lambda m: m.datetime)

        return JSONResponse([item.format() for item in sorted_messages])

    @basicauth(realm=REALM)
    @throttle(10,1*60,"PostMessage")
    def POST(self, request):
        """
        Parses the JSON request to create a Message object
        """
        item = Message()
        item.parse(request)
        if not item:
            return http.HttpResponseBadRequest()

        item.save()

        return http.HttpResponse(status=StatusCodes.CREATED)

    @basicauth(realm=REALM)
    def DELETE(self, request):
        """
        Deletes all the messages on the database (must be a staff user)
        """
        if request.user.is_staff:
            Message.objects.all().delete()
            return http.HttpResponse(status=StatusCodes.DELETED)
        else:
            response = http.HttpResponseForbidden()
            return response

All the methods requires basic authentication, so you'll need to register an user with the registration api, so then you can then provide the authentication for the messaging api.

The GET method accepts GET parameters in the URL and returns the latest 50 messages related to the user making the request.

You can specify the to_user parameter, if you want to fetch the messages of a specific conversation, and a datetime parameter, if you want to have a minimum date settled.

Example of a GET request:

api/messaging/?to_user=Bruno&datetime=2010-12-12 04:00:00

I created an utility class, JSONResponse, subclassing HttpResponse

class JSONResponse(http.HttpResponse):
    """
    Creates an instance of HttpResponse containing JSON content
    """
    def __init__(self, data):
        indent = 2 if settings.DEBUG else None
        mime = ("text/javascript" if settings.DEBUG 
                                  else "application/json")
        super(JSONResponse, self).__init__(
            content = simplejson.dumps(data, indent=indent),
            mimetype = mime,
        )

It's used to create a json response based on python dictionary objects, and lists of dictionary objects.

In MessagingApi GET case it passes a list of formatted Messages to JSONResponse, that outputs them in the content of the response in JSON format.

The POST method accepts a JSON input rapresenting a message, it parse the request to a Message object model, and saves it to the datastore. If it can't parse the request it returns Bad Request status, if everything is ok, it returns the status Created

Example of a POST Request:

{ "to_user": "God", "message": "Hi God!" }

The DELETE method checks if the user is part of the staff (you can modify an user by going into the Datastore Viewew in the SDK Console, or typing the url: _ah/admin/datastore/), if it is, it deletes all the messages from the datastore, and returns status Deleted.

I hope you'll find this useful for your REST knowledge.

Update (08/02/2011)

I updated the source to include the latest release of django-nonrel, djangoappengine and django-toolbox

Browse on GitHub