Ansbile 1.8 and Boto

Are you getting this Ansible error message, even though you are damn sure boto is installed?

msg=’boto required for this module’

One possible reason for this is the message is referring to the status of the remote machine, not the machine that you are running Ansible from and you do not have boto installed on the remote machine. Doh!

Also, by default, Ansible uses the default python interpreter. This is probably not your virtualenv (if you use one). Boto needs to be installed for that interpreter. You can check to see if this is the problem, by SSH-ing to the remote machine and running:

sudo pip install boto

 

 

 

Advertisements

Redirecting Webfaction Subdomain to AWS

I just started playing around with AWS. I have a domain name that is served by Webfaction. I wanted to create a subdomain and point it at my AWS EC2 instance. Seems like it should be pretty easy. But the time it takes for domain name propagation makes makes it hard to debug.

The first thing to note is the command:

host domain_name.com

lets you know if propagation has been done.

CNAME

The first approach I tried was to set the CNAME for the subdomain using the Webfaction control panel. The was super easy, but it did not work. It appears AWS will not allow that. For more info, see here.

.htaccess

The other approach is to create a static site on Webfaction and use .htaccess to redirect. Something like this. One advantage of this approach is once the domain name has propogated, you do not need to wait for it to re-propagate if you change .htaccess.

If you get a 500 error, you can inspect the Webfaction logs to find out what went wrong. The log you want is ~/logs/apache/error_my_app_php.log.

Here is the htaccess file that worked for me:

Redirect permanent / http://ec2-xx-xx-x-xxx.compute-1.amazonaws.com
order deny,allow

 

 

 

 

Troubleshoot Django, Gunicorn and Nginx

Recently I was trying to get a Django site running on AWS using Gunicorn and Nginx. The problem I ran into was page loads were very slow – about 30 seconds. I have created some pretty non-optimized Django sites, but have never seen page load times greater than 5 seconds. Clearly something was wrong. Here are my notes for solving this problem. First, I would like to give a big shout-out to the folks on Reddit. It was with their help that I was able to solve this.

This problem required a divide and conquer approach. The ultimate problem being how to divide Django, Unicorn and Nginx when there were no error messages pointing to any one of them. But let me start with some of the things I did leading up to that.

One easy test was to ping the AWS server instance that I created. Ping gave a time of about 60 msec. So the request was getting to the server.

Most Django sites use a separate server for static content, like images or CSS files. You can have a slow web-page, if the static server is slow and your web-page uses static content. The easiest way to check if thats the problem is using the profiling tools available for most browser. These tools will show you a time line of the content the browser is loading to render the page.

To further test Django, you can create a template that has no context variables and no static content. Mine looks like this:

<!DOCTYPE html>
<html>
<head lang="en">
    <meta charset="UTF-8">
    <title></title>
</head>
<body>
<p>Test page</p>
</body>
</html>

Then I add this to my urls.py:

url(r'^test_page/$', TemplateView.as_view(template_name='test_page.html'))

This page should be blazing fast. If all this runs well locally, then its time to get it going on your remote server.

One problem I constantly run into is getting all my paths right. Especially the one to wsgi.py. For this particular problem, I put some logging into wsgi.py. But for some reason, even though wsgi.py was being run, I was not getting the logging file. I suspect the reason was that my problem was causing a system time out which prevented the logging file from being written. I am not sure this is the problem. But I use Python logging a lot, and this is the first time  did not get the output I expected. However, along the way I found an easy way to determine if wsgi.py is being run; delete wsgi.pyc, if it reappears when you restart the server, you know it was run.

It’s also easy to have a problem with settings. For that just run the django runserver command and see  if it kicks off errors on startup.

I did this and it still consistently took 30 seconds for each page load. It turns out that seemingly innocuous fact is significant; it turns out that it is the default time-out time for many programs. That fact that it never varied is the clue. This was suggested by a few people on Reddit. Lonely_b really nailed it. OK, so its a time-out. But what is timing out.

The next step in the debugging process was inspired by LibidinousIntent on Reddit. He/she suggested serving a simple plain HTML file directly from Nginx by adding another location. Turns out I already had such a file – my 500.html file. Here is a snippet of the code in my Nginx-app-proxy file:

  error_page 500 502 503 504 /500.html;
  location = /500.html {
    root /home/my_project/templates;
  }

I could test this aspect of nginx with the URL:

http://mysite.com/500.html

In my case, the 500 page displayed instantly (no 30 second load time), indicating that Nginx was working. So is it Gunicorn or Django?

santagada on Reddit suggested using python simplehttpserver to by-pass Gunicorn. I implemented this idea by:

  1. SSH-ing into the server.
  2. Turning off Gunicorn
  3. CD-ing into the manage.py directory
  4. Running Django runserver with the port Nginx was using

In the terminal window, I could see the Django development server messages as I went to the test page on the site. Thirty seconds after the request, I got an error traceback related to a time out. Several of the messages complained about a cache problem. In the development version I was using DummyCache. In the AWS version, I was using Django-elasticache. I changed the AWS version to DummyCache and the load time problem disappeared! What a relief. Thanks Reddit!

Installing Askbot on AWS

AWS is a moving target. Some docs are over 4 years old. Below are my notes. Before you read further, I would like to point out that I abandoned this approach because I found Elastic Beanstalk to be too frustrating.

AskBot is an open source, Django based question and answer platform, similar to Stackoverflow. Here are some useful links:

What sets my install apart from those is I want to use Django 1.5 and I want to use Postgresql.

If you scan through the AWS guide, you will see that it requires Ruby in addition to Python. That seems messed up. The purpose of this is to run the eb command line tool. If you look at the page for the eb tool, as of 2014-04-21 Ruby is no longer required.

Install Elastic Beanstalk CLI (eb command)

Download the source from here. Easy enough. But the install instructions in README.txt tell you to add “eb” to your path. Problem is there are several eb files. What worked for me is to add the dir that contains the eb file you want. I use Python 2.7, so my path command was:

export PATH=$PATH:/home/AWS-ElasticBeanstalk-CLI-2.6.2/eb/linux/python2.7

CD into repo root. Run:

eb init

Most of the answers to the questions you will get are pretty straight forward. One that might throw you is environment tier. Choose

WebServer::Standard::1.0

Then run

eb start

Chris Gagne said that, among other things, eb start would create the folder .ebextensions in the repo root. This did not happen for me, so I created it manually. I added the file askbot.conf using the content here as a template.

I continued to follow Chris Gagne’s blog through using aws.git to post the code. The odd thing was my apps were not showing up on the AWS console. Turns out I had the console pointing at the wrong region of the world.

Several Hours Later

I am giving up on EB. I am going to try the approach given by Ashok Fernandez.