Independent Django Apps, Migrations and Version Control

The Django docs recommend keeping project migrations in version control. This makes sense. Let say you are modifying a model in development. You get it working. Run your tests, etc… and now you are ready to put it into production. If your migrations are in version control, then to migrate production, all you have to do is push the new code, along with the migrations, to the production server and run Django migrate. Pretty simple and fool-proof.

But what if you are creating a re-usable Django app. When I develop a re-usable app, I put it in a simple Django project for testing during development and as a demo of the app. During the process, I often create migrations for the app. Should those migrations be in version control?

If your apps are all independent of each other, it does not matter. However, if your apps are not independent, then using the migrations can lead to problems. In my case, I have two apps: User Management (UM) and Events. Events have foreign keys to users. Thus they are not independent. The problem that arises is related to a model in UM that contains the field: models.FilePathField.

That field has a required attribute called path that is that absolute path to the folder to get choices from. During dev, it gets set to a string for a path for the sample project. This is hard-coded in. When you install the app in a different project and run migrate, migrate sees that the path is no longer correct and creates a migration to correct the path. The problem comes when I create a project that uses both these apps. The version of UM in version control does not have the migration that Events needs. This causes the project migration to fail.

One possible solution is to hack the offending migration. That’s not a very good solution because it would require hacking every time a new version of the app is uploaded to the project. It also is possible that the hack might not be done right, which could make a mess.

The solution I settled on was to the MIGRATION_MODULES setting to create app migrations inside the project. This will also ignore the migrations in the app package. But that’s what you really want anyway. The project migrations are for migrating the project database. The only reason the app migrations work is because often the project db and the app db are similar.

If you use this approach, be aware that you will need to run makemigrations for each app in MIGRATION_MODULES setting. Because it is not always obvious which apps need to be in MIGRATION_MODULES, I have created a custom command to set things up:

import os

from django.core.management.base import BaseCommand
from django.core.management import call_command

from django.conf import settings


def make_package(root):
    if os.path.exists(root):
        return

    os.mkdir(root)
    fp = open(os.path.join(root, '__init__.py'), 'wb')
    fp.close()


class Command(BaseCommand):
    """
    For putting all installed app migrations into a migrations folder in this project.

    This command also creates a settings file called migration_modules_settings.py. After that it
    runs initial migrations on each app. Finally it runs the migrate command.

    Before running this command, make sure your settings file has something like:

        try:
            from migrations.migration_modules_settings import MIGRATION_MODULES
        except:
            pass

    """
    help = 'Sets up project migrations and runs migrate.'

    def handle(self, *args, **options):
        manage_py_path = os.getcwd()
        migrations_dir = os.path.join(manage_py_path, 'migrations')
        make_package(migrations_dir)

        # For creating a settings file
        migration_modules = {}

        for full_app_name in settings.INSTALLED_APPS:
            app_name = full_app_name.split('.')[-1]
            root = os.path.join(migrations_dir, app_name)
            make_package(root)

            migration_modules[app_name] = 'migrations.' + app_name

        fp = open(os.path.join(migrations_dir, 'migration_modules_settings.py'), 'w')
        fp.write('MIGRATION_MODULES = {\n')
        for key in migration_modules:
            fp.write("    '{}': '{}',\n".format(key, migration_modules[key]))
        fp.write('}\n')
        fp.close()

        call_command('makemigrations', *migration_modules.keys())
        call_command('migrate')

Problems with the PyCharm Test Runner

I had some problems with the PyCharm test runner. This was strange since I could run the tests from the command line. It turns out the problem was due to the project root being pretty far down the list in sys.path. The PyCharm test runner was picking up migrations in other apps. Adding this code to the top of my settings file solved the problem:

import sys
sys.path.insert(0, 'PATH CONTAINING MIGRATION FOLDER')

 

Cannot Runserver After Django CircularDependencyError

Lets say you attempt a Django 1.7 migration and you get this error:

django.db.migrations.graph.CircularDependencyError

Well that sucks.

In my case, the model change was of minor importance and I did not have time to figure out how to resolve the circular dependency. So I reverted all changes and moved on. But now “runserver” fails with the circular dependency error. Somehow the error persists after the code is reverted. How does this happen?

Turns out that makemigrations creates a folder called “notmigrations” in the folder that contains your project. Delete this folder and you will be back to the initial state.

 

Django Pluggable Apps, Migrations and Version Control

The goal: a plugable Django app, with migrations, where the migrations are stored in the repo of the plugable app. At first this might seem impossible because a plugable app does not have manage.py. Luckily for me, I always include a bare-bones django project in a folder called “example”. This project uses the plugable app. This means the migration code will be in the app repo.

The first step is to make the initial migration for the app:

python manage.py makemigrations my_app

after this command, you should see a migrations folder in your plugable app folder. Add those files to version control.

Next, update the plugable app in your projects and for each project, run migrate.

Django Makemigrations Wants to Remove an Existing Field

Django migrations are awesome. But even the docs say you should inspect the proposed migrations before running “migrate”. Recently I ran “makemigrations” and Django wanted to remove a field that was still in my model. What’s going on here?

In this case, the model had lots of fields and added @property methods. It turns out that there was a @property method that had the same name as a model field. I will not go into details about how this happened. Anyway, it turns out that the @property method was an error. Removing it and deleting the migration file fixed the problem.

Django Migrations: “No changes detected”

If you get the message “No changes detected” after running:

python manage.py makemigrations MY_APP

AND you have seen all the Q&A on Stackoverflow related to first migrations or migrating from an older version AND none of that applies to you then may be I have the answer.

Maybe your model code has a bug. In my case, I had a complex and expensive calculation that I added as a @property of my model. Later, I decided to slightly de-normalize my database and actually make that property a database field. The problem was I forgot to delete the @property method. This was over-riding my new field. Which is why it was not detected. Duh…