Using Mock with Django

Django has lots of great tools for testing. All of them involve creating a test database before the tests are run. This can be slow.

Python Unittests are another important part of testing. They often are very fast. But what if you want to do unit tests that involve Django model instances? This is where mock comes in. Mock lets you create objects that function like model instances without constructing a database. This Celerity Blog post is a great way to get started.

One problem you may encounter while using mock.patch is this error:

AppRegistryNotReady: Apps aren't loaded yet.

This can be solved by adding this to your test:

import django
django.setup()

You might need to setup some paths and specify a settings file. I do not have to do that because I run the tests from within PyCharm and it does some of that stuff automatically.

Here is a sample of how you might test a Django request that accesses a Django model instance:

import unittest
import json

from django.http import JsonResponse

import mock

def get_contract_items_for_ticket(request):
    """
    AJAX function for loading contract item select based on job
    :param request: Django request object
    :return:
    """
    # Import here to avoid circular imports
    from data.models import Job

    job = Job.objects.get(id=int(request.POST['ticket_job_id']))
    ci_choices = get_contract_items(job.number)
    response = {'choices': ci_choices}
    return JsonResponse(response)


class MyTest(unittest.TestCase):
    import django
    django.setup()

    def setUp(self):
        # Setup code here

    def test(self):
        job_w_contract = mock.Mock()
        job_w_contract.number = '2222-22'
        job_w_contract.id = 22

        job_wo_contract = mock.Mock()
        job_wo_contract.number = '1111-11'
        job_wo_contract.id = 1

        # Function for mocking the .get() method
        def get_job(id=None):
            if id == job_w_contract.id:
                return job_w_contract
            else:
                return job_wo_contract

        # Mock the import statement and test the function
        with mock.patch('data.models.Job') as my_model_mock:
            my_model_mock.objects = mock.Mock()  # mocks Job.objects

            # Make Job.objects.get() use get_job()
            conf = {'get.side_effect': get_job}
            my_model_mock.objects.configure_mock(**conf)

            # Make request object for a job with contract items
            mock_request = mock.Mock()
            mock_request.POST = {'ticket_job_id': str(job_w_contract.id)}

            # Do a request that returns choices
            response = get_contract_items_for_ticket(mock_request)
            choices = json.loads(response.content)['choices']
            self.assertTrue(len(choices) > 1)

            # Do a request that does not return choices
            mock_request.POST = {'ticket_job_id': str(job_wo_contract.id)}
            response = get_contract_items_for_ticket(mock_request)
            choices = json.loads(response.content)['choices']
            self.assertTrue(len(choices) == 1)

Module Level Global Variables in Python

Global variables are bad. Great! Got that out of the way.

Here is why I needed a global variable. I had a module that wrote a bunch of files and directories to a directory. I wanted to do unit tests and during unit tests, I wanted to write everything to a test directory. A global variable seemed like a good way to do that, but it’s trickier than you might think. For the details, see SO.

This approach does NOT work:

from unittest import TestCase

TESTING = False

def make_path():
    if TESTING:
        path = 'xyz'
    else:
        path = 'abc'
    return path


class MyTest(TestCase):
    def setUp(self):
        TESTING = True

    def test(self):
        self.assertEqual(make_path(), 'xyz')

It does not work because the line “TESTING = True” in the test setup because it creates a new local variable called TESTING. The global TESTING remains False.

One approach that does work is this:

from unittest import TestCase

TESTING = {'use_testing_path': False}

def make_path():
    if TESTING['use_testing_path']:
        path = 'xyz'
    else:
        path = 'abc'
    return path


class MyTest(TestCase):
    def setUp(self):
        TESTING['use_testing_path'] = True

    def test(self):
        self.assertEqual(make_path(), 'xyz')

In this case, the test setup function does not create a new variable. Instead it modifies the attribute of the existing global variable.

Migrating django_manage from Ansible 1.9 to Ansible 2.5

Here is one of the errors (to help you get here from Google):

"msg": "\n:stderr: /usr/bin/env: python\r: No such file or directory\n"

I had an old Django manage.py file that worked for Ansible 1.9. It did not have a shebang (e.g. #!/usr/bin/env python). The error it was generating was:

"msg": "[Errno 8] Exec format error", "rc": 8

Reading the docs, I found this key tidbit:

As of ansible 2.x, your manage.py application must be executable (rwxr-xr-x), and must have a valid shebang, i.e. “#!/usr/bin/env python”, for invoking the appropriate Python interpreter.

That seemed easy enough. But I was not sure what shebang to use since I was using a virtualenv. The quickest way to find out seemed to be use nano to edit manage.py on the server, run the Ansible task that was failing. Repeat until it works.

Then things got weird. Every reasonable shebang lead to the error I started this post with. Also, for no obvious reason, nano started warning me about the file being in DOS format, which is strange since this project has always been in Linux. Maybe when I was cutting and pasting some shebangs into the file I ended up with some DOS whitespace. Running the command “dos2unix manage.py” fixed it.

The shebang that worked was:

#!/usr/bin/env python

UPDATE

It appears that the DOS format chars entered manage.py a long time ago. Maybe when the project was created. I ended up fixing the file in my repo.

 

Logging a Big Process

Lets say you have a web site. When the user clicks a link it runs a process that generates a huge report, lots of ins and outs. Lots of places where some of the data might be questionable, but not bad enough to give up. What you really want to do is warn the user. The problem is your code is pretty modular. You could pass around a variable to keep track of the issues, but wouldn’t it be better if there were a more unified approach? Some sort of error accumulator… maybe a logger. Wait that’s built in to Python. This works:

# other_module.py
import logging
logger = logging.getLogger('my logger')

def f3():
    logger.debug('test f3')

and

import logging
try:
    from cStringIO import StringIO      # Python 2
except ImportError:
    from io import StringIO
    
import other_module

logger = logging.getLogger('my logger')
logger.setLevel(logging.DEBUG)
logger.propagate = False
formatter = logging.Formatter('%(module)s.%(funcName)s:%(lineno)d - %(message)s')
log_stream = StringIO()
handler = logging.StreamHandler(log_stream)
handler.setFormatter(formatter)
logger.addHandler(handler)


def f1():
    logger.error('test f1')


def f2():
    logger.debug('test f2')


def complex_process():
    # Clear stream to limit errors to each call to main
    log_stream.seek(0)
    log_stream.truncate()

    f1()
    f2()
    other_module.f3()
    errors = log_stream.getvalue()
    print(errors)

complex_process()
complex_process()

Results in :

logging_example.f1:21 - test f1
logging_example.f2:25 - test f2
other_module.f3:8 - test f3

logging_example.f1:21 - test f1
logging_example.f2:25 - test f2
other_module.f3:8 - test f3

Warning: do not use logging.basicConfig() for this. As stated in the docs “it’s intended as a one-off simple configuration facility, only the first call will actually do anything: subsequent calls are effectively no-ops.” If you do use this function, it is likely it will not do anything and the logger you will end up using is the root logger. One big problem with that is you will receive logging messages from lots of other, unexpected modules, such as third party modules.

Receive Text Messages on a Website Using Twilio

In a nutshell:

  1. Setup and get a phone number on Twilio
  2. Configure Twilio
  3. When some one sends a text to the phone number, Twilio packs info about the text in an HTTP POST and posts to your website at the URL you provided when configuring Twilio

Here is a sample of the POST:

{
    u'Body': [u'Hello world!'],
    u'MessageSid': [u'SMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'],
    u'FromZip': [u'53706'],
    u'SmsStatus': [u'received'],
    u'SmsMessageSid': [u'SMxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'],
    u'AccountSid': [u'AC63444b3f5817d72cbbadb35a71bdd2e9'],
    u'FromCity': [u'MADISON'],
    u'ApiVersion': [u'2010-04-01'],
    u'To': [u'+16089999999'],
    u'From': [u'+16081234567'],
    u'NumMedia': [u'0'],
    u'ToZip': [u'53703'],
    u'ToCountry': [u'US'],
    u'NumSegments': [u'1'],
    u'ToState': [u'WI'],
    u'SmsSid': [u'SMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'],
    u'ToCity': [u'MADISON'],
    u'FromState': [u'WI'],
    u'FromCountry': [u'US']
}

That should get you started. See the Twilio documentation for details.

SSH Stops Working After Ubuntu Upgrade

I recently upgraded from 14.04 to 16.04. It went pretty smoothly, except some of my SSH connections stopped working. It turns out the the upgrade, automatically upgraded OpenSSL from version 6 to version 7 and version 7 no longer allows keys that it thinks are insecure.

To see what version you are running:

ssh -V

The big problem is the server I need to connect to is managed by a “Windows” guy who hates Linux. Getting him to update the key is going to take a while and I need to connect NOW.

The solution is at: http://www.openssh.com/legacy.html

I put this in my  ~/.ssh/config file:

Host XXX.XXX.XXX.XXX
    HostKeyAlgorithms +ssh-dss

It’s ugly, but it works.