Troubleshoot Django, Gunicorn and Nginx

Recently I was trying to get a Django site running on AWS using Gunicorn and Nginx. The problem I ran into was page loads were very slow – about 30 seconds. I have created some pretty non-optimized Django sites, but have never seen page load times greater than 5 seconds. Clearly something was wrong. Here are my notes for solving this problem. First, I would like to give a big shout-out to the folks on Reddit. It was with their help that I was able to solve this.

This problem required a divide and conquer approach. The ultimate problem being how to divide Django, Unicorn and Nginx when there were no error messages pointing to any one of them. But let me start with some of the things I did leading up to that.

One easy test was to ping the AWS server instance that I created. Ping gave a time of about 60 msec. So the request was getting to the server.

Most Django sites use a separate server for static content, like images or CSS files. You can have a slow web-page, if the static server is slow and your web-page uses static content. The easiest way to check if thats the problem is using the profiling tools available for most browser. These tools will show you a time line of the content the browser is loading to render the page.

To further test Django, you can create a template that has no context variables and no static content. Mine looks like this:

<!DOCTYPE html>
<html>
<head lang="en">
    <meta charset="UTF-8">
    <title></title>
</head>
<body>
<p>Test page</p>
</body>
</html>

Then I add this to my urls.py:

url(r'^test_page/$', TemplateView.as_view(template_name='test_page.html'))

This page should be blazing fast. If all this runs well locally, then its time to get it going on your remote server.

One problem I constantly run into is getting all my paths right. Especially the one to wsgi.py. For this particular problem, I put some logging into wsgi.py. But for some reason, even though wsgi.py was being run, I was not getting the logging file. I suspect the reason was that my problem was causing a system time out which prevented the logging file from being written. I am not sure this is the problem. But I use Python logging a lot, and this is the first time  did not get the output I expected. However, along the way I found an easy way to determine if wsgi.py is being run; delete wsgi.pyc, if it reappears when you restart the server, you know it was run.

It’s also easy to have a problem with settings. For that just run the django runserver command and see  if it kicks off errors on startup.

I did this and it still consistently took 30 seconds for each page load. It turns out that seemingly innocuous fact is significant; it turns out that it is the default time-out time for many programs. That fact that it never varied is the clue. This was suggested by a few people on Reddit. Lonely_b really nailed it. OK, so its a time-out. But what is timing out.

The next step in the debugging process was inspired by LibidinousIntent on Reddit. He/she suggested serving a simple plain HTML file directly from Nginx by adding another location. Turns out I already had such a file – my 500.html file. Here is a snippet of the code in my Nginx-app-proxy file:

  error_page 500 502 503 504 /500.html;
  location = /500.html {
    root /home/my_project/templates;
  }

I could test this aspect of nginx with the URL:

http://mysite.com/500.html

In my case, the 500 page displayed instantly (no 30 second load time), indicating that Nginx was working. So is it Gunicorn or Django?

santagada on Reddit suggested using python simplehttpserver to by-pass Gunicorn. I implemented this idea by:

  1. SSH-ing into the server.
  2. Turning off Gunicorn
  3. CD-ing into the manage.py directory
  4. Running Django runserver with the port Nginx was using

In the terminal window, I could see the Django development server messages as I went to the test page on the site. Thirty seconds after the request, I got an error traceback related to a time out. Several of the messages complained about a cache problem. In the development version I was using DummyCache. In the AWS version, I was using Django-elasticache. I changed the AWS version to DummyCache and the load time problem disappeared! What a relief. Thanks Reddit!

Advertisements

One thought on “Troubleshoot Django, Gunicorn and Nginx

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s