Making an SSH known_hosts File for Ansible

When I use Ansible, I often use SSH to download files from multiple locations. A recurring problem is Ansible hangs because it is waiting for someone to accept the host’s SSH finger print. The solution is to use ssh-key to make the known_hosts file. Here is the gist:

- name: Create a known hosts file for root
  shell: ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts

 

Advertisements

SSH, Private Repos, Ansible and Vagrant

Lets say you want to setup a server using Vagrant and Ansible and you want to put your code onto the server by checking it out from a private repo from some place like BitBucket. Further, you want to make it possible for the server to checkout new updates. Here is one way to do it.

First create a RSA keypair. There are lots of examples of how to do it. This is a good one. When you complete this, you will have the keypair on your local machine, which will be useful if you want to check code from the repo.

Next, put a copy of your public key (*.pub) on the repo site (e.g. Bitbucket). The repo site will have instructions for how to do this.

After that, create an Ansible YMAL file and put both the private and public keys in it. Something like this:

---
git_repo: ssh://git@bitbucket.org/me/my_repo.git

- name: Create the Bitbucket SSH public key file
  copy: src="/path/to/your/local/public/key"
        dest=/home/vagrant/.ssh/id_rsa_bitbucket.pub
        mode=0644
        owner=vagrant
        group=vagrant

- name: Create the Bitbucket SSH private key file
  copy: src="/path/to/your/local/private/key"
        dest=/home/vagrant/.ssh/id_rsa_bitbucket
        mode=0600
        owner=vagrant
        group=vagrant

- name: Setup the Git repo git: repo={{ git_repo }} dest={{ project_path }} accept_hostkey=yes when: setup_git_repo tags: git

When you run vagrant up or vagrant provision, you might get this error:

Permission denied (publickey).
 fatal: The remote end hung up unexpectedly

However, if you run vagrant ssh and manually git the repo, everything works. The reason for this is, in my case, Ansible was becoming root before pulling the code from the repo and root does not have the private key. Luckily, you can tell Ansible git which file to use, by using the key_file keyword. Like this:

- name: Setup the Git repo
  git: repo={{ git_repo }} dest={{ project_path }} accept_hostkey=yes key_file=/home/vagrant/.ssh/id_rsa_bitbucket
  when: setup_git_repo
  tags: git

Django, PyCharm, Vagrant and Ansible

To provide maximum isolation between customers, I prefer to set each one up on its own VPS. One down-side to this approach is it is a pain to setup a new server. In the past, I used Fabric. But recently I read High Performance Django. In that book, they go out of their way to advise against using Fabric. They like Salt. After a little googling and a few coin tosses, I decided to give Ansible a try.

At first the going was a little rough, then I discovered this gem of a repo on Github: ansible-django-stack. It even uses Vagrant, another package I have wanted to try.

OMG

This repo is insanely helpful. It even comes with a fully functional Django project. This was incredibly helpful for figuring out how to get Pycharm working with the virtual machine created by Vagrant. Just follow the docs in repo and in a few minutes you will have a fully functioning Django site, running the components any serious Django project needs. This will change your life.

Installing the Exact Same Configuration on Digital Ocean

Getting it running on a local virtual machine is pretty cool. Its even more amazing how easy it is to get it running in a Digital Ocean droplet. In a previous post, I tried and failed to setup a droplet using Ansible. So I skipped that step and use the Ansible web interface to do that. Here are the steps for creating the droplet and installing the complete stack:

  1. Setup a Droplet using Ubuntu 12.04.5×64 because that is what the Vagrantfile specifies. If you want to setup a different OS, change your Vagrantfile.
  2. Get the IP address for your new droplet
  3. cd into the root dir of ansible-django-stack and type:

ansible-playbook -i ip_address, -v development.yml --ask-sudo-pass

Make sure to include the comma after the IP address. This tells Ansible to not look for an inventory file and to just run the playbook on that server. That’s it! Go to http://192.168.33.15/ and you will see a fully functioning Django site!

Bring the Charm

One powerful feature of Vagrant is it’s “Synced Folders” function. Actually this is a misnomer. A synced folder is a folder that is accessible on your host machine and the virtual machine. Thus there is no synching involved. They are the same physical location.

You can use this feature to edit project files “on” the virtual machine from the PyCharm installation on the host machine. To activate folder synching, do the following:

  1. git clone https://github.com/jcalazan/glucose-tracker.git   /home/me/glucose-tracker
  2. Add this line to the Vagrantfile in ansible-django-stack:
config.vm.synced_folder "/home/me/glucose-tracker", "/webapps/glucosetracker/glucose-tracker"

Now open this repo with PyCharm just like you would with any other project. If you change some of the code and restart the server on the virtual machine, you will see the changes when you reload the page.

Yeah But Can I Use the PyCharm Debugger?

The best feature of PyCharm is the debugger. But will it work with Vagrant? Of course it will. Those folks at Jet Brains have got you covered.

PyCharm (v4) fully supports Vagrant and can run the python interpreter on the virtual machine. To get started, just go to settings>tools>vagrant and do what comes naturally. Then go to Tools on the menu bar and select vagrant up.

Next, go to settings>Project>Project Interpreter. Click the gear icon and select Add Remote, then select Vagrant and set the instance folder to the folder that contains Vagrantfile. Next set the python interpreter to the virtualenv for this project (note how awesome having a working project is):

/webapps/glucosetracker/bin/python

Next go to Run>Edit Configurations to setup the PyCharm server. Here is what it looks like:

run_config

You will need to setup these environment vars:

env_vars

Finally, you need to setup the Path Mappings like this:

path_mappings

And ta da! You can now run the project in PyCharm complete with debugging.

Now you can add the power of Ansible and Vagrant without sacrificing any of the power of PyCharm!

Creating a Digital Ocean Droplet using Ansible

***Warning: I took these notes as I tried to create a droplet using Ansible. I did not get it to work. So if all you want is the answer, this is not the place to be.

I have been using AWS free tier for a while. Based on lots of googling and the kindness of other bloggers, I eventually got a fabric script that sets up and manages my sites on AWS. Getting that fabric script working was a real challenge because the AWS API is a moving target and the docs are lacking.

I just found out my free period on AWS is about to end. While I think AWS is probably a pretty good way to go if you have a lot of traffic and VC’s paying your bills, many of my sites are very low volume (50 users, less than one page view per second). Thus AWS is expensive over-kill.

As I have been googling various subjects, I noticed that lots of answers are on Digital Ocean. So I started looking into Digital Ocean as an alternative to AWS. Everything seemed great, so I signed up and started playing around.

I should also mention that several months ago, I read “High Performance Django“. In that book, they have a special, highlighted warning to not use fabric for deployment. Silly me. Just because fabric works well and is easy to understand is no reason to use it. After more googling, I decided to try Ansible.

My first impression is that if you stick to the well beaten path on Digital Ocean, things work pretty good. I did their one click installer for Django and had a site running in less than a minute. If you are a noob to Django and deployment, this is the way to go. But I wanted more control of my stack and there seems to be lots of Ansible examples for creating a Django stack on DO. Down the rabbit hole I went.

Turns out DO and Ansible are moving targets with lacking documentation. AWS all over again.

This Github repo seemed to hold the answer: https://github.com/hostmaster/ansible-digitalocean-bootstrap. But right off the bat, I got:

failed=True msg='dopy >= 0.2.3 required for this module'

I had dopy 0.3 installed. But Ansible was not using my virtualenv. I add the virtualenv by adding this line to vars.yml:

ansible_python_interpreter: "/home/me/.virtualenvs/roi_digital_ocean/bin/python"

With that fixed, I am stymied by this error:

failed: [localhost] => {"failed": true}
msg: Access Denied

I posted it to Stackoverflow. No answer in over 48 hours. I am not complaining. In fact I am grateful for the free help I have gotten in the past. However, in contrast, the last Django question I asked was answered in 10 minutes. I suspect this question is obscure enough that I will never get an answer.

I know Ansible uses dopy. So I tried creating a droplet with dopy. Following the examples, I was able to create a droplet. However, I was not able to create a droplet using the ssh_key_ids keyword in the new_droplet method. I tried every conceivable value. Every time I got:

dopy.manager.DoError: You specified invalid ssh key ids for Droplet creation.

Which is the error message the Digital Ocean API sends back. Here is a solution to a similar problem. They are using the python Requests module. dopy uses Requests. The difference between that solution and dopy is dopy does not explicitly JSON encode the parameters. I think Requests does that for you if you give it a dict. Therefore, in theory, there is no difference between the two. Also, I was able to create a new droplet, with an SSH key using python-digitalocean v1.1.

For now, maybe I can just use the DO web interface to create the droplets and use Ansible to configure them once they are created. Here are the gory details for Ubuntu 12.04. I already have an SSH key on my machine:

  1. Go to the .ssh dir on your machine. Mine is at ~/.ssh
  2. Open id_dsa.pub and copy its entire contents
  3. Goto: https://cloud.digitalocean.com/ssh_keys
  4. Click on Add SSH Key
  5. Paste the contents of id_dsa.pub into the Public SSH Key box
  6. At the end of the SSH key you will see a name. I used that as the name on the DO web form. I am not sure if this is necessary
  7. Use the DO web interface to create a new droplet. In the Add SSH Key section, make sure to highlight the SSH key you just made.
  8. I use ssh from the Ubuntu command line all the time. So it made sense to test my new droplet there with:
ssh root@104.236.77.94

Once More with Ansible

In the “getting started” section of Ansible, it shows you how to use Ansible to ping a server. Lets do that.

  1. create an inventory file. I called mine test_inventory. It’s contents was just the IP address of my droplet.
  2. In the same dir as the inventory file, at the command line, type:
ansible -i test_inventory -m ping all -u root

If you forget the -u root params, you will get a message like:

SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv ...

If you run it again with the -vvvv option, you get an very confusing and long set of debug messages. This is the problem with Ansible. Simple mistakes that noobs are likely to make, lead to confusing error messages. I am sure the “Access Denied” error above is a similar noob mistake.