Daemonizing Django-RQ using Supervisor

I was trying to set up a task to be run from Django-RQ. The task involved scrapping a webpage using Selenium and Google Chrome. It worked great in development, but not in production. The error message indicated that there were problems starting Chrome.

One big difference between dev and production was in production I was daemonizing Django-RQ using Supervisor. Some queued tasks would run. Just not the ones involving Selenium. The clue came when I stopped Django-RQ using supervisorctl and then started it from the command line. Now the Selenium tasks worked.

I solved the problem by adding this snippet to the top of the task that used Selenium:


import os
import json

json.dump(os.environ['PATH'].split(':'), open('debug_file.json', 'wb'))

This revealed that the environment PATH when running from the command line was much different that that when running Django-RQ from Supervisor. Adding some of those paths to the Supervisor config solved the problem.

[program:django_rq]
command= {{ virtualenv_path }}/bin/python manage.py rqworker high default low
stdout_logfile = /var/log/redis/redis_6379.log

numprocs=1

directory={{ django_manage_path }}
environment = DJANGO_SETTINGS_MODULE="{{ django_settings_import }}",PATH="{{ virtualenv_path }}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
user = vagrant
stopsignal=TERM

autostart=true
autorestart=true

Django HTTP 404 Not Found Error

So you’ve got everything working on the Django dev server. Almost everything is working on the staging server… except one URL that sometimes fails with an HTTP 404 Not Found error. What is going on here?

More details; it happens to be a Django form page. The page loads with a problem. The problem occurs after you submit the form. The URL that causes the 404 error is the URL of the form. What is going on here?

In my case, the form_valid() was doing a lot of processing of the form data and the server was timing out. Maybe part of the reason I did not catch this earlier is my dev server is much faster than the Digital Ocean droplet I was using.

Django Long Running Processes

The most commonly suggested solution for long running processes is to use Celery. I suspect that if you need scalabilty or high volume, etc… Celery is the best solution. That said,  I have been down the Celery rabbit hole more than once. It has never been pleasant. Since my needs are more modest, maybe there is a better alternative?

My needs involve running process that might run for 15 minutes or so. The process might run a dozen times/day and be launched by as many users. The process must be launch-able from the website by authorized users.

I have solved this problem by going to the polar opposite of Celery – cron. Every minute cron would launch a custom django command. That command would look in a database table for tasks and get input data from the database. When a task was completed, that fact was written to the database table. Honestly, this approach has worked well. Never-the-less, I always wonder if there is a stable, simple, robust solution that lies somewhere between cron and Celery.

Maybe RedisRQ and Django RQ? These are my notes so that a year from now, when this issue comes up again, I can get up to speed quickly.

Step 1: Install Redis and Start Redis Server

These instructions are pretty good.

Step 2: Is Redis Server Working?

Maybe you installed Redis Server a long time ago and you want to see if it’s still working? Go here.

Or you could type:

$ redis-cli ping
PONG

Step 3: Install RQ

pip install rq

Step 4: Install and Configure django-rq

Go here.

Step 5: Read the RQ Docs

Seriously – read the RQ docs. They are brief and to-the-point.

Step 6: Daemonize the RQ Workers

If you use supervisord , here is the Ansible template I use to do that:

[program:django_rq]
command= {{ virtualenv_path }}/bin/python manage.py rqworker high default low
stdout_logfile = /var/log/redis/redis_6379.log

numprocs=1

directory={{ django_manage_path }}
environment = DJANGO_SETTINGS_MODULE="{{ django_settings_import }}",PATH="{{ virtualenv_path }}/bin"
user = vagrant
stopsignal=TERM

autostart=true
autorestart=true


[program:rqscheduler]
command={{ virtualenv_path }}/bin/python manage.py rqscheduler
stdout_logfile = /var/log/redis/rq_scheduler.log

numprocs=1

directory={{ django_manage_path }}
environment = DJANGO_SETTINGS_MODULE="{{ django_settings_import }}",PATH="{{ virtualenv_path }}/bin"
user = vagrant
stopsignal=TERM

autostart=true
autorestart=true

Reflections

I do not recall all the problems I had with Celery. After reviewing the RQ solution above, it is clear that one of the advantages of that solution is the documentation is really good. Or at least it clearly and directly addressed what I was trying to do.

Additionally, I wish I would have implemented this a long time ago. It is so easy to use. And it’s so freeing to to be able to run long processes.

Reflections part Deux, Troubleshooting and Gotchas

It’s coming back to me. The supervisor config in the original post started the daemon OK. But it turns out there was an error in the config that caused the queued processes to fail. Finding and fixing that bug was a pain in the ass. Maybe my troubles with Celery were really troubles with supervisor? Down the rabbit hole we go!

It turns out that the Django RQ Queue Statistics are helpful for debugging. They show failed tasks along with a Python traceback! Very nice. In my case, I was getting the error:

ImportError: No module named XXXX

Clearly one of my paths in the supervisor conf file was wrong. Time to start hacking:

  1. Edit conf file
  2. Run supervisorctl stop django_rq
  3. Run supervisorctl start django_rq
  4. Queue a task
  5. It failed again? How is that possible? Back to step 1

GOTCHA! After a while you notice the changes you are making are not having any effect. And then you recall that to reload the config file you must run:

service supervisor restart

Now my config file works. All I have to do is figure out which of the ever cludgier hacks I made can be removed. The config file above has been updated.

Son of Reflections part Deux – Adding PATH to Supervisor Config

I thought I had it working. Then when I added a slightly more complex task that interacted with the database, it failed with an ImportError. After flailing around for a while, I found that adding a PATH to the supervisor environment variable solved the problem.

During my flailing, I found this blog post. Lots of great ideas.

Still Falling Down the Rabbit Hole – Logging to the Rescue

Everything was working almost every where… except with the daemonized workers on the server. Luckily, Django-RQ now comes with logging. I implemented the logging in the Django settings files as per the docs, restarted the dev server, and… no logging. Turns out you have to restart the workers.

Also, although the docs show the use of a logging class made for rq (rq.utils.ColorizingStreamHandler), it turns out you can use logging.FileHandler, which is what you want for debugging the code when running from a daemonized worker.

For what it’s worth, it turns out the problem was with the python locale module. The docs say something about it not being thread safe. The function locale.getlocale() returned a value when the workers were run via the dev server, but it returned None when run from a daemonized worker.

Datepickers for Django Sites that Work on the Desktop and Mobile

As of April 2017, HTML datepickers are still a mess. Here’s how it should be; all browsers should support the <input> attribute “type”. And when the browser sees <input type=”date” …. > the browser should offer up an awesome builtin date picker. Unfortunately we are far from that world.

My work around starts with using a javascript datepicker plugin. There are lots of choices and I have not tried them all. The one I am using is Bootstrap Datetime Picker. When using this picker, make sure to set <input type=”text” …. >, by doing this you prevent the browser from simultaneously providing a native date picker. For example, when you are running in “no icon” mode, with type=text this is what you get in Chrome:

but when type=”date” you get this:

Pretty ugly. It’s even worse in IE. This works pretty well on big screens, but not as well on small screens (e.g. phones).

On the small screen, it’s best to use the native datepicker. To make that happen you need not initialize the datepicker widget and you need to change the type attribute to “date”. Here is one way to do that that uses the Responsive Bootstrap Toolkit:

(function($, viewport){
    $(document).ready(function() {
        var datepickers = $(".bootstrapdatepickerwidget3");

        // Executes in XS and SM breakpoints
        if(viewport.is('>=md')) {
            datepickers.each(function(index, el) {
                $(el).datetimepicker({format: 'YYYY-MM-DD'});
            })
        } else {
            datepickers.each(function(index, el) {
                $(el).attr('type','date');
            })
        }
    });
})(jQuery, ResponsiveBootstrapToolkit);

Add a custom Django form field widget and you can make all these steps automatically:

from django import forms

class BootstrapDatePickerWidget(forms.DateInput):
    # noinspection PyClassHasNoInit
    class Media:
        css = {'all': [
            '3s_hts/js/bootstrap-datetimepicker-master/bootstrap-datetimepicker.min.css'
        ]}

        js = ('3s_hts/js/bootstrap-datetimepicker-master/moment.js',
              '3s_hts/js/bootstrap-datetimepicker-master/bootstrap-datetimepicker.min.js',
              '3s_hts/js/responsive-toolkit/dist/bootstrap-toolkit.min.js',
              '3s_hts/js/setup_bootstrap_datepicker.js')

class BootstrapDatePickerField(forms.DateField):
    widget = BootstrapDatePickerWidget3

Grouping Choices in a Django Select

To make an ordinary Django ChoiceField you can do something like this:

choices = [[1, 'Apples'], [2, 'Oranges'], [3, 'Carrots'], [4, 'Beans']]
my_choices = forms.ChoiceField(choices=choices)

But what if you want to group the choices by using the HTML OPTGROUP tag? My first thought was to over-ride some on the methods in the Django Select widget code. But when I inspected the code, I found the code already supports this (Django 1.8). Here is how to do it:

choices = [
    ['Fruit', [[1, 'Apples'], [2, 'Oranges']]], 
    ['Veggies', [[3, 'Carrots'], [4, 'Beans']]]
]
my_choices = forms.ChoiceField(choices=choices)

Django, Nginx, Invalid HTTP_HOST header

From time to time, I get an email from my Django server complaining about “Invalid HTTP_HOST” and looking like some hacker is trying to mess with my gunicorn sock. Here is the relevant part of the email:

Invalid HTTP_HOST header: u'/webapps/MY_SITE/run/gunicorn.sock:'. The domain name provided is not valid according to RFC 1034/1035.

Request repr():
<WSGIRequest
path:/arlo/,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'HTTP_ACCEPT': '*/*',
 'HTTP_CONNECTION': 'close',
 'HTTP_USER_AGENT': 'masscan/1.0 (https://github.com/robertdavidgraham/masscan)',
 'HTTP_X_FORWARDED_FOR': '120.88.152.6',
 'PATH_INFO': u'/arlo/',
 'QUERY_STRING': '',
 'RAW_URI': '/arlo/',
 'REMOTE_ADDR': '',
 'REQUEST_METHOD': 'GET',
 'SCRIPT_NAME': u'',
 'SERVER_NAME': '/webapps/MY_SITE/run/gunicorn.sock',
 'SERVER_PORT': '',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'SERVER_SOFTWARE': 'gunicorn/19.6.0',

How is this hacker getting the path to my gunicorn sock?

It turns out this is happening because I am using $http_host in my nginx config file:

upstream wsgi_server {
 server unix:/webapps/MY_SITE/run/gunicorn.sock fail_timeout=0;
}

server {
 listen 80;
 server_name MY_DOMAIN_NAME;

 location / {
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header Host $http_host;
     proxy_redirect off;
     if (!-f $request_filename) {
        proxy_pass http://wsgi_server;
        break;
     }
 }

when a request is made to the server and the HTTP Host is empty, nginx sets the HTTP host to the gunicorn sock.

I can generate this error using curl:

curl -H "HOST:" MY_DOMAIN_NAME -0 -v

This sends a request without a HTTP Host. The -0 causes curl to use HTTP version 1.0. If you do not set this, the request will use HTTP version 1.1, which will cause the request to be rejected immediately and not generate the error.

The solution is to replace $http_host with $host (as pointed out on Stackoverflow). When HTTP Host is missing, $host will take on the value of the “server_name” directive. This is a valid domain name and is the one that should be used.