Django Long Running Processes

The most commonly suggested solution for long running processes is to use Celery. I suspect that if you need scalabilty or high volume, etc… Celery is the best solution. That said,  I have been down the Celery rabbit hole more than once. It has never been pleasant. Since my needs are more modest, maybe there is a better alternative?

My needs involve running process that might run for 15 minutes or so. The process might run a dozen times/day and be launched by as many users. The process must be launch-able from the website by authorized users.

I have solved this problem by going to the polar opposite of Celery – cron. Every minute cron would launch a custom django command. That command would look in a database table for tasks and get input data from the database. When a task was completed, that fact was written to the database table. Honestly, this approach has worked well. Never-the-less, I always wonder if there is a stable, simple, robust solution that lies somewhere between cron and Celery.

Maybe RedisRQ and Django RQ? These are my notes so that a year from now, when this issue comes up again, I can get up to speed quickly.

Step 1: Install Redis and Start Redis Server

These instructions are pretty good.

Step 2: Is Redis Server Working?

Maybe you installed Redis Server a long time ago and you want to see if it’s still working? Go here.

Or you could type:

$ redis-cli ping
PONG

Step 3: Install RQ

pip install rq

Step 4: Install and Configure django-rq

Go here.

Step 5: Read the RQ Docs

Seriously – read the RQ docs. They are brief and to-the-point.

Step 6: Daemonize the RQ Workers

If you use supervisord , here is the Ansible template I use to do that:

[program:django_rq]
command= {{ virtualenv_path }}/bin/python manage.py rqworker high default low
stdout_logfile = /var/log/redis/redis_6379.log

numprocs=1

directory={{ django_manage_path }}
environment = DJANGO_SETTINGS_MODULE="{{ django_settings_import }}",PATH="{{ virtualenv_path }}/bin"
user = vagrant
stopsignal=TERM

autostart=true
autorestart=true


[program:rqscheduler]
command={{ virtualenv_path }}/bin/python manage.py rqscheduler
stdout_logfile = /var/log/redis/rq_scheduler.log

numprocs=1

directory={{ django_manage_path }}
environment = DJANGO_SETTINGS_MODULE="{{ django_settings_import }}",PATH="{{ virtualenv_path }}/bin"
user = vagrant
stopsignal=TERM

autostart=true
autorestart=true

Reflections

I do not recall all the problems I had with Celery. After reviewing the RQ solution above, it is clear that one of the advantages of that solution is the documentation is really good. Or at least it clearly and directly addressed what I was trying to do.

Additionally, I wish I would have implemented this a long time ago. It is so easy to use. And it’s so freeing to to be able to run long processes.

Reflections part Deux, Troubleshooting and Gotchas

It’s coming back to me. The supervisor config in the original post started the daemon OK. But it turns out there was an error in the config that caused the queued processes to fail. Finding and fixing that bug was a pain in the ass. Maybe my troubles with Celery were really troubles with supervisor? Down the rabbit hole we go!

It turns out that the Django RQ Queue Statistics are helpful for debugging. They show failed tasks along with a Python traceback! Very nice. In my case, I was getting the error:

ImportError: No module named XXXX

Clearly one of my paths in the supervisor conf file was wrong. Time to start hacking:

  1. Edit conf file
  2. Run supervisorctl stop django_rq
  3. Run supervisorctl start django_rq
  4. Queue a task
  5. It failed again? How is that possible? Back to step 1

GOTCHA! After a while you notice the changes you are making are not having any effect. And then you recall that to reload the config file you must run:

service supervisor restart

Now my config file works. All I have to do is figure out which of the ever cludgier hacks I made can be removed. The config file above has been updated.

Son of Reflections part Deux – Adding PATH to Supervisor Config

I thought I had it working. Then when I added a slightly more complex task that interacted with the database, it failed with an ImportError. After flailing around for a while, I found that adding a PATH to the supervisor environment variable solved the problem.

During my flailing, I found this blog post. Lots of great ideas.

Still Falling Down the Rabbit Hole – Logging to the Rescue

Everything was working almost every where… except with the daemonized workers on the server. Luckily, Django-RQ now comes with logging. I implemented the logging in the Django settings files as per the docs, restarted the dev server, and… no logging. Turns out you have to restart the workers.

Also, although the docs show the use of a logging class made for rq (rq.utils.ColorizingStreamHandler), it turns out you can use logging.FileHandler, which is what you want for debugging the code when running from a daemonized worker.

For what it’s worth, it turns out the problem was with the python locale module. The docs say something about it not being thread safe. The function locale.getlocale() returned a value when the workers were run via the dev server, but it returned None when run from a daemonized worker.

Django, Celery and Memory Use

Disclaimer: These are the notes of a Celery/RabbitMQ/Linux noob. Not sure how generalizable they are.

A lot of the Django projects I develop have a small number of users and run pretty light-weight tasks. Thus server costs SHOULD be pretty low. However, they also frequently involve asynchronous, “long-running” tasks, such as creating reports, etc… One common way to handle these tasks is with Celery and RabbitMQ.

For my needs, Celery and RabbitMQ are massive overkill. Despite that, I use them because they are well documented and very stable. There are two promising alternatives:

Neither one is beyond v 0.5. In other projects, I have used a combination of cron and custom django commands. That has worked well, but it does not seem as flexible as something like Celery.

The biggest problem with Celery and RabbitMQ is memory use. For my basic Django project, running all by itself in Vagrant, the command “ps aux”, shows these memory values in percent:

  • django – 6%
  • gunicorn – 3%
  • supervisor – 3%
  • postgres – 6 processes – 6%
  • rabbitmq beam.smp – 8%
  • other rabbitmq – 4 processes – negligible
  • celery beat – 8%
  • celery workers – 3 processes – 24%

This is with no tasks running or web requests coming in. It seems crazy to me to devote 40% of memory to Celery and friends. Maybe I can do some tuning.

I am not sure why I have 3 Celery workers. I am guessing one is enough. Here is my command to start Celery:

celery worker -A my_app -E -l info --concurrency=2

this is what gave 3 processes. When I set concurrency to 1, I got 2 processes. I also got two processes with:

celery worker -A my_app -E -l info --autoscale=3,1

OK, so it looks like I will need to accept 16% for celery workers. Maybe I can improve things by switching to Redis?

I installed Redis using the quick start guide and the defaults. Redis memory usage is negligible. Should you switch to Redis? It depends. There are other important differences. Google RabbitMQ vs Redis. For many of my projects, memory use trumps the other differences.

Celery 3.1, Django and Webfaction

Despite wanting to keep things as simple as possible, occasionally I run into circumstances where I need my Django app to be able to run long processes. While there are lots of different ways to hack this together, Celery offers a clear and well tested option. These are my notes on the install and config process on Webfaction.

About Celery

What is Celery? Celery is a python module that does three things. It provides decorators to make python functions into executable tasks. It provides methods for putting those tasks in to a message queue. It also provides a daemon that monitors the message queue and executes tasks. The message queue is a separate piece of software, that runs in the background, for example RabbitMQ or Redis.

Configuring Redis

The Celery docs seem to prefer RabbitMQ over Reddis. Here is what they say about Redis “… is more susceptible to data loss in the event of abrupt termination or power failures.” However RabbitMQ is based on Erlang, so in addition to installing RabbitMQ, you have install Erlang.

Redis is based on GCC and libc, so the install is really easy. Since my use case could easily tolerate abrupt termination, I chose Redis. The quickstart covered everything I needed to get things working on my Ubuntu 12.04 development machine.  This tutorial is also pretty good. It’s not necessary to read, but its pretty concise and contains a lot of useful information that may give you ideas of how to use Redis outside of Celery.

To get Redis running on Webfaction, check out this post on Stackoverflow. Most notably, you will need to use Webfaction’s control panel to create a custom app listening on port.

Configuring Celery

Here are the Celery docs. Prior to Celery version 3.1, the Django app for interacting with celery separate from celery. In version 3.1, much of the Django functionality is built in. Keep that in mind if you run into problems and google around for solutions.

I am using Redis to store the results, so there is no need to install django-celery.

I am putting celery in my virtualenv. To install, activate your virtualenv and run:

pip install celery

Do not do the general “First steps with Celery” section. It shows a way to configure celery that is somewhat different from how the docs show to setup celery for Django. No doubt you could reconcile the approaches, but who needs that? Instead jump to the celery Django section. For now, ignore the suggestion to read the “First steps” section.

One thing that gave me lots of grief is my Django project has the “old” structure where manage.py is in the same dir as settings.py. This messed up the relative and absolute imports. To solve this, in the root of my Django project, I renamed my module celery.py to my_celery.py. And I changed __init__.py to reflect this. I also removed the project name from settings to become:

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings')

Redis uses a pre-defined default port number. If you use all the defaults in development, things go pretty smoothly. The problem is, you cannot use that same port on Webfaction. The best solution is to run redis in development using the same port you will use on Webfaction. This will make it more likely to work when you move it to Webfaction. To start redis on a port:

redis-server --port 13789

Next, you need to add some settings to Django settings:

BROKER_URL = 'redis://127.0.0.1:13789/0'
CELERY_RESULT_BACKEND = BROKER_URL

Now cd into your django project root and run:

celery -A my_celery worker -l info

Pay attention to the info celery displays when it starts. You should see your settings in the “transport” and “results”. You should also see your celery tasks under “[tasks]”

Once you get everything running, its time to use supervisor to launch celery as a daemon.

Supervisor

To make all this work, you need to manage some daemons. The consensus for that is you need to run supervisor. The docs are pretty good. I used pip to install.

pip install supervisor

I installed it outside of my virtualenv so that it could be available to all my virtualenvs. Make sure to read the section “Creating a Configuration File“. It is short and to the point. In the celery section “[program:celeryworker]” make sure to point the command to your virtualenv:

command=/home/me/.virtualenvs/my_virt/bin/celery worker -A my_celery --loglevel=INFO

To activate your virtualenv, point your PATH environment variable to the bin directory of your virtualenv:

environment=PATH=/home/me/.virtualenvs/my_virt/bin

This does the same thing that the virtualenv activate command does.

Also set directory to point to the root of your project:

directory=/home/me/3s_hts ; directory to cwd to before exec (def no cwd)

Once its installed and configured, you can run it with:

supervisord

To shutdown supervisor first stop all processes with:

supervisorctl stop all

Then kill supervisor with a normal kill command.

When I start celery this way, I get 9 processes devoted to celery. For my needs this is over-kill. To get fewer processes change the “command” line in supervisord.conf to:

command=/home/athena/.virtualenvs/qdb6/bin/celery -A my_celery --concurrency=3 worker

Gotchas

Adding Tasks

So you are working in development mode. You fired up Celery worker and start writing some tasks in your Python/Django project. Of course, you realize that the python code is auto-detecting the tasks, but no big deal because the development server restarts when you save your new task. Then you go to run it and you get:

“Received unregistered task of type”

It turns out that each time you add a task, you need to restart Celery. If you look carefully at the messages as Celery loads, you will see a list of tasks that it has found.

Restarting Celery

If you use supervisorctl to stop celery, it will NOT stop the celery workers. To do that, you need a command like:

ps auxww | grep 'celery worker' | awk '{print $2}' | xargs kill -9

This is from the celery docs. I tried various supervisor settings, such as such as stopsignal, and killasgroup, but none of them stopped the workers. I posted this solution on Stackoverflow. You might want to check that out to see if someone has come up with a better solution.