Using the Ansible Find Command

The “find” command was added to Ansible in version 2. Here is how I used it to change the permissions on some log files:

- name: Get all log files in django manage.py dir
  find:
    paths: "{{ django_manage_path }}"
    patterns: "*.log"
    recurse: yes
  register: files_to_change
    

- name: Make sure Django log files in manage.py dir are owned by vagrant
  become: yes
  file: path={{ item.path }} owner=vagrant group=admin mode=0660
  with_items: "{{ files_to_change.files }}"
Advertisements

Ansible Could Not Find Templates After Migration from 1.9 to 2.5

I had a playbook that included a task from another role like this:

tasks:
  - include: roles/django/tasks/create_server_settings.yml

This include stopped working when I migrated from Ansible 1.9 to 2.5. The task used the “template” command and Ansible could not find the template. It looked in:

  • roles/django/tasks/templates/server_settings.py.j2
  • roles/django/tasks/server_settings.py.j2

I am using the recommended directory structure with the tasks and templates directories both at the same level in the directory tree, in this case:

  • roles/django/templates/server_settings.py.j2

Switching to the command:

 tasks:
  - include_tasks: roles/django/tasks/create_server_settings.yml

did NOT help.

The solution was to use this command:

tasks:
  - include_role:
      name: django
      tasks_from: create_server_settings

Monitoring Django RQ

In my use case, users used a form to put a long running process on the queue. I wanted to make a page that would allow each use to see the status of the jobs they queued. This turned out to be slightly more difficult than it should be.

The first step involved saving the job information to the user’s session:

from datetime import datetime

from django.conf import settings
from django.views.generic import FormView
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse

import pytz
import django_rq

class MyView(FormView):
    def form_valid(self, form):
        queue = django_rq.get_queue('low')
        job = queue.enqueue(
            generate_report_func,
            int(form.cleaned_data['year']),
            email_list=form.cleaned_data['recipients']
        )

        # Save queued job to session as a list so that order is preserved
        now = pytz.timezone(settings.TIME_ZONE).localize(datetime.now())
        enqueued_jobs = self.request.session.get('enqueued_jobs', [])
        enqueued_jobs.append({
            'job_id': job._id,
            'name': 'Revenue and Open POs Report {}'.format(form.cleaned_data['year']),
            'started': now.isoformat(),
            'started_for_display': now.strftime('%Y-%m-%d %H:%M'),
            'queue': 'low'
        })
        self.request.session['enqueued_jobs'] = enqueued_jobs
        self.request.session.modified = True
        return HttpResponseRedirect(reverse('on_queue'))

 
Here is the code for the view that allows the user to see the status of each job:

import django_rq
from redis import Redis
from rq.registry import StartedJobRegistry

from django.views.generic import TemplateView

from dateutil.parser import parse


class OnQueueView(TemplateView):
    template_name = 'on_queue.html'

    def get_context_data(self, **kwargs):
        kwargs = super(OnQueueView, self).get_context_data(**kwargs)

        # Make a list of queued jobs
        queue = django_rq.get_queue('low')
        job_status = {x._id: x.status for x in queue.jobs}

        # Make a list of running jobs
        redis_conn = Redis()
        registry = StartedJobRegistry('low', connection=redis_conn)
        for job_id in registry.get_job_ids():
            job_status[job_id] = 'running'

        # Make a list of failed jobs
        for job_id in django_rq.get_failed_queue().job_ids:
            job_status[job_id] = 'failed'

        # Insert status into list of jobs, remove old jobs
        all_jobs = self.request.session.get('enqueued_jobs', [])
        kwargs['jobs'] = []
        now = pytz.timezone(settings.TIME_ZONE).localize(datetime.now())
        self.request.session.modified = False
        for job in all_jobs:
            dt = (now - parse(job['started'])).total_seconds()
            if dt < 3600 * 4:
                if job['job_id'] in job_status:
                    job['status'] = job_status[job['job_id']]
                else:
                    job['status'] = 'completed'
                kwargs['jobs'].append(job)
                self.request.session.modified = True

        if self.request.session.modified:
            self.request.session['enqueued_jobs'] = kwargs['jobs']

        return kwargs

Django Sessions in StaticLiveServerTestCase

This is for Django 1.8.

Here was the scenario. I have a Django site where most of the pages require the user to login. I have a form that gets some initial values from Django sessions. The form has lots of javascript, so live testing seemed best. The problem I could not set the session values in the test code.

Initially, this seemed pretty easy because in the test code, there was a variable called:

self.client.session

However, changes to the session were not present when the test code called the view. Careful inspection showed that the session id’s in self.client.session and view.request.session were different.

Adding the following method to StaticLiveServerTestCase solved the problem:

    def save_to_session(self, key, value):
        cookies = self.selenium.get_cookies()
        session_key = None
        for cookie in cookies:
            if cookie[u'name'] == u'sessionid':
                session_key = cookie[u'value']
                break

        if session_key:
            from django.contrib.sessions.backends.cached_db import SessionStore
            s = SessionStore(session_key)
            s[key] = value
            s.save()

 

Django and Pytest

As of Django 1.8, when you run tests and have migrations, the tests build the test database from the migrations (if you do not believe me, run tests with -v 3). If you have a lot of migrations, this can be painfully slow.

Pytest has an option that builds the test database directly:

pytest --nomigrations ./my_test.py

In my case, I get about a x5 speed up.

This works for  django.test.TestCase and django.contrib.staticfiles.testing.StaticLiveServerTestCase.

Unfortunately, pytest is not a drop in replacement for django test.

Failed: Database access not allowed, use the “django_db” mark

Sometimes you will get the above error. Based on the error message, it seems like you may need to alter your code some how. However, according to the docs, if you are using django.test.TestCase you do not need to alter your code.

One way you get get this error is if your imports access the database. Move the imports to inside the test class and the problems go away.

Python Mock Functions Not Being Called

Are you using mock to mock some functions during testing and mysteriously your tests are ignoring the mock decorator?

One way this can happen is if the function being mocked is either defined in the module that is being tested or is imported into the module being tested. How to get around this?

In my case, after I wasted time figuring out why mock was not working, I ended up modifying the code in my module so that I did not need mock. In the end, I think the modified code is better anyway. Often making code testable, makes it better too.