Django-Webtest Tips and Gotcha’s

Based on lots of comments and other stuff on the “Internets”, I have started using Django-Webtest. Webtest inherits from Django TestCase. It allows you to interact with forms more like the user does. For example you can select an option from an HTML Select tag.  I am still trying to decide if this extra functionality is worth having to deal with another layer of documentation. Especially since that layer is sparse.

Anyway, these are my notes for getting things done with Webtest. This post is not meant to be a tutorial or replace the docs. Rather its just stuff I have found useful and hard to find in the docs.

Viewing the Page in a Browser

Let’s say you are testing a form and things are not going as planned. Sometimes the easiest way to solve the problem is to view the page. Here is how:

form = self.app.get(reverse('login')).form
form['email'] = user.email
form['password'] = self.test_password
response = form.submit().follow()
response.showbrowser()

The last line is where the magic is.

Finding Stuff on the Page

Webtest provides the Beautiful Soup parsed HTML in the .html attribute of a response. One way this is useful is to assert things about the content of a page. For example:

response = self.app.get(reverse('my_url', args=[obj.id]), user='jim')
delete_button = response.html.find('button', id='delete_button')
self.assertIsNotNone(delete_button)

Checking the URL After Redirect

Sure, you can use status code. But what if you want to know where you were redirected? Use this Django assertion:

response = form.submit()
self.assertRedirects(response, reverse('my_named_url'))
response = self.app.get(response.url, user='test')  # do the redirect

Occasionally this does not work. For some reason, the port gets inserted in the response.url, while its not in reverse(). I have seen this problem occur with some assertRedirects and not others in the same test method. Here is one way to get around this:

from urlparse import urlparse

def my_assertRedirects(self, response, expected_url):
    parts = urlparse(response.url)
    self.assertEqual(parts.path, expected_url)

Handling MultipleChoiceField

I am using django-crispy forms. When I make an InlineCheckBoxes of a Django MultipleChoiceField, I end up with several HTML checkbox tags with the same name. Hence the usual method for assigning a value does not work. Here is something that does work:

form.set('my_checkbox_name', True, index=1)

where the index keyword specifies which checkbox with name “my_checkbox_name”.

Handling Form Errors

One way to handle form errors is to look at the response_status. For most of my forms, a status of 200 is returned when there is an error. While a 302 (redirect) is returned when there are no errors. Although this works, it feels like a pretty blunt instrument for handling form errors.

One interesting fact about WebTest is it inherits from Django TestCase. What his means is that some of the functionality of TestCase still works. For example, you can submit a form using the post method of the Django Client class.

For error handling, you can still use:

response = form.submit('save')
self.assertFormError(response, 'form', 'user_type', u'This field is required.')

or for non field errors:

response = response.form.submit('save')
self.assertFormError(response, 'form', None, ['an error message', 'another error message'])

Note that the word form is a string. It is the name of the form in the context dict that you send to render. Here are the docs on assertFormError.

What if you are getting a form error, but you are not sure what error it is? The place to start looking is in the response.context list.

Form Submit

Lets say you have a form that redirects after it is successfully submitted. If you submit it like this:

response = form.submit('save')
print response.status
>>>302 FOUND
response.showbrowser()

In the browser, you will see a blank page.  What is going on here is Webtest did not automatically go to the page you are redirecting to. To have it go to that page, do this:

response = form.submit('save').follow()

ModelForm Initial Values

This one is baffling. If you pass initial values to a ModelForm using the initial keyword, they do not appear in the form! How do I know this? If I run the code outside of testing, they appear. If I run the code in testing and do showbrowser() they are not there. If I set break points during testing, I can see that the “initial” keyword has the correct values.

HTTP Forbidden (403)

With the systems I design, it is often necessary to limit access to views based on attributes of the user. Thus a big part of testing is making sure users cannot view things they are not allowed to view.

You would think this would be a simple test:

response = self.app.get(reverse('my_view'))
self.assertEqual(response.status_code, 403)

But if you do that you get an error:

AppError: Bad response: 403 FORBIDDEN

Instead try this:

response = self.app.get(reverse('my_view'), status=403)

If the response status is not 403, an error will be raised, which is exactly what you want.

Clicking a Link

If there is a link on your page (<a></a>) the webtest response object has a method called “click” which lets you click on the link. Seems easy enough. You can send regular expression patterns or strings to select the link you need.

The gottcha is when the uses “onclick” as its action. When the method is searching for the link, it also checks to see what the tags href attribute is. If there is no href attribute, then the link is skipped even if it matches the linkid.

In a way this makes sense because the click method ends with a call to the goto method. In any case, WebTest does not run javascript. Guess I need to use selenium for this.

Use the Verbose Keyword

Several methods accept the keyword “verbose”. Use it to speed up debugging.

Sessions

The good news is you can inspect session variables by looking at:

self.app.session

The bad news is I cannot figure out how to set a session variable inside a test. It seems that writing to self.app.session does not work. WebTest uses a special backend to handle auth. It seems that is not connected to the normal session processing. If you know how to do this, let me know!

Ugh… this worse than I thought. For some mysterious reason, the session vars are not carrying over between views. I am solving this by switching to Django TestCase.

Testing A Mobile Django Site on a Phone on a LAN

Getting a site to layout well on mobile can be tricky. Although there are some decent mobile emulators, I still frequently run into important differences when I view it on a real phone.

Web pages for the desktop are easy to test using the Django development server. This note describes how to browse the development server from a mobile device that is on the same LAN as the server.

    1. Connect to the LAN with your phone
    2. Get the local IP address of the computer that runs the development server. On Linux the command is ifconfig (that’s not a typo. It’s not ipconfig). My computer’s IP address is 192.186.1.64
    3. Start the development server as follows: python manage.py runserver 0.0.0.0:8800 
    4. Start a browser on your phone
    5. Goto: http://<the server IP address>:8800  In my case that would be http://192.168.1.64:8800

You should see your website on your phone now.

Can’t Connect?

Is wi-fi on on your phone?

 

 

 

Mobile Detection on Django

Most of the Django sites I develop these days need to work on mobile. I am using Bootstrap 3 to make my pages responsive. Bootstrap’s philosophy is “mobile first”. But there are many cases where I do not want users with big screens to be penalized by a site that is optimized for a small screen.

While Bootstrap has some css classes for showing and hiding based on screen size. That approach is somewhat limited. It also slows the page down because the content still needs to be downloaded even if its not shown. A more complete solution would be to have the option of detecting the device on the server side and altering the page before its sent. This is especially easy using Django templates.

The big question is how to detect the users device and hopefully get its screen size in Django. I started with fiftyonedegrees.mobi. I was able to install it on Ubuntu 12.04 without any problems. I was using the “lite-pattern-wrapper” method, so I did not download the trie database. It worked fine, locally with the Django development server. However, when I tried to install it on a CentOS 5, 32-bit machine (Webfaction), it seg faulted. The folks at Webfaction suggested that I migrate to one of their CentOS 6, 64-bit machines. It installed without problems on the machine. But when I tried using the code by fetching a page from my DroidX, it seg faulted. I am not sure how to debug something like that. It works locally. And when it seg faults there are no debugging messages.

Wait. All is not lost. The middleware and context processor works with Firefox on Ubuntu. I added a bunch of print statements to the middleware. The problem occurs in the call to mobile_detector.match in _match. Up until then all the parameters look reasonable. To make matter worse, if I fire up my virtualenv and python interpreter on the server and run that command from the command line, with the params that caused the seg fault, everything works fine. It looks like this is a problem with mod_wsgi.

There is some stuff on stackoverflow related to this. I tried adding WSGIApplicationGroup %{GLOBAL} but no change. This is a mess. Moving on. I am going to try a pure python package.

django-mobile looks OK, but seems to be limited to just differentiating between mobile, tablet and full. Moble ESP looks interesting. The python code is just one file: mdetect.py. It’s very readable and understandable. Not as many Django features as django-mobile. But most of what I want, I can roll my own. Here is quick and dirty middleware:

 
from xxx import mdetect

class DetectMobile:
    def process_request(self, request):
        user_agent = request.META.get("HTTP_USER_AGENT")
        http_accept = request.META.get("HTTP_ACCEPT")
        if user_agent and http_accept:
            agent = mdetect.UAgentInfo(userAgent=user_agent, httpAccept=http_accept)
            request.mobile_esp_agent = agent   # in case we want more information about the device
            if agent.detectMobileQuick():
                request.device_type = 'mobile'
            elif agent.detectTierTablet():
                request.device_type = 'tablet'
            else:
                request.device_type = 'desktop'
        else:
            request.mobile_esp_agent = None
            request.device_type = 'desktop'   # default

The careful reader will no doubt notice that after I called django-mobile limited, I set about creating those exact limits in my middleware. Ha. But an even carefuller reader will notice I include the agent in request, so I can get at the details as needed. So far this approach is working well.

 

Django Modelform with Many-to-Many

Django (1.5) Modelform is supposed to handle many-to-many fields. In my case, I was editing the Django User model with Django groups. Everything seemed to be working correctly. The correct group associations were automatically showing up in the many-to-many widget. The problem was the new associations were not being saved.

A quick look at the docs revealed a discussion of the problems that can happen with many-to-many when a save is done using commit=False. But I was not doing that.

Turns out the problem was in my multi-select widget. I am using the “Whitelabel” theme from revaxarts.com.It auto-magically replaces clunky widgets with better widgets. When I used a bare-bones HTML template, the many-to-many worked.

When I looked at the cleaned data right before the save command, I noticed that the Groups query set was empty. Adding the following to the form’s clean methods solved the problem:

def clean_groups(self):
    if 'groups[]' in self.data:
        group_ids = [int(x) for x in self.data['groups[]']]
        g = Group.objects.filter(id__in=group_ids)
    else:
        g = Group.objects.none()
    return g