Start a real world Django project with virtual environment

Updated on November 2016

There are some great posts on how to setup a new virtual environment and/or a real world Django project. However, the framework and Python itself evolves. Also, every project is unique and has it's own requirements. Thus, the setup workflow depends on your language and library versions and needs.

These are my notes for Django project setup workflow for Python 3.4 (or later) and Django 1.10.

Before following these steps you should have Python 3.4 or later installed on your machine.

Virtual environment

A virtual environment might sound complicated but it's basically just a Python project specific directory, which contains a Python executable and the Python packages you choose to install in the virtual environment.

When you start developing a new Python application, large or small, I strongly recommend creating a virtual environment for the project. The most important reason for using a virtual environment is that with virtual environments you can have multiple separate Python projects on the same machine using different set or versions of packages or different versions of Python itself. Using a virtual environment also allows you to neatly save all 3rd party library references of your project (with nothing extra) and install the same versions on your production server. More on this later in this post under pip.

Default Python binary

Your operating system probably has one or more versions of Python already installed and in your path. You can check your default Python version and the location of the binary with

$ which python
/usr/bin/python

$ python -V
Python 2.7.10  

In my case, the default Python binary is located in /usr/bin/python (to be precise, this location is likely a symbolic link to the real binary) and it is version 2.7.10. This Python binary has it's own built-in and 3rd party python modules/packages that it uses. The default, machine wide, packages for this python version might be in a path like /usr/local/lib/python2.7/site-packages/.

We are not going to use this Python nor the modules/packages. Instead we create a separate project specific python binary and place to install packages, i.e. we create a virtual environment.

Python 3 venv module

For creating a virtual environment you have two options. Virtualenv or venv. Virtualenv is a separately installed Python package which works with Python versions from 2.6 to 3.4. Venv on the other hand is shipped with Python, but it's only available with Python 3.3 and later.

Both work very similarily. I use venv when possible because then there is one less package to install.

Another tool for handling virtual environments is conda, which is popular in scientific domain.

Virtual environment setup

First you must create a virtual environment for your project (once per machine you use to develop or deploy your project). You can choose where the virtual environment directory is located. Two common patterns are the following:

1. Separate directories for all projects and all environments.

|-code
   |
   |-projects
   |     |-facetagram
   |     |-instatube
   |
   |-virtualenvs
         |-facetagram
         |-instatube

2. Virtual environment directory under each projects directory

|-code
   |
   |-facetagram
   |    |- <project code dirs and files>
   |    |-facetagramenv
   |    
   |-instatube
        |- <project code dirs and files>
        |-instatubeenv

The virtual environment directory is per machine, and can be in different path (including dir name) relative to project dir on different machines for the same project. Usually the virtual environment directory, i.e. binaries, is not saved to version control.

If you have Python 3.4 installed on your machine, you should have pyvenv-3.4 on your path.
If you follow the first example pattern for virtual env dir, create a new virtual environment with

$ cd /your/path/to/code
$ pyvenv-3.4 virtualenvs/demoproject

You can also do the same thing by calling

$ cd /your/path/to/code
$ python3 -m venv virtualenvs/demoproject

This -m <module> syntax can be used, because venv is a built in module in python (version 3.3 forwards).

Now your folder structure should be like this:

|-code
   |
   |-projects
   |     |-facetagram
   |     |-instatube
   |
   |-virtualenvs
         |-facetagram
         |-instatube
         |-demoproject

Using virtual environment

Once you have created the virtual environment, you must activate the environment you want to use. If no virtual environment is activated, you will use the default Python and packages.

Activate a virtual environment

$ source virtualenvs/demoproject/bin/activate
(demoproject)$

After a successful activation you should see the name of the active virtual environment (the name of the virtual env folder) in your terminal prompt. You can stop using a virtual environment with command deactivate. Note that the activate is a bash script that adds the deactivate function to your shell when you run it. If you can't find deactivate, it might be because you haven't run activate first and thus deactivate does not exist in your shell session.

Check that you are now using the Python from your virtual environment, not your machine default.

(demoproject)$ which python
/your/path/to/code/virtualenvs/demoproject/bin/python

(demoproject)$ python -V
Python 3.4.1

(demoproject)$ deactivate

$ which python
/usr/bin/python

Note that which python returns a different value when a virtual environment is active compared to when the virtual environment is deactivated.

Autoenv (optional)

Do this after you have created your Django project.

It can be hard to always remember to activate and deactivate your environment when you start or stop working on your project. To help with this, I use autoenv. Check autoenv install instructions from the Github link.

After you have autoenv installed, you can create a .env file in any directory. The content of this file will be executed whenever you cd in that directory. An .env file is a very practical thing to place in your project base dir with the virtual environment activation command in it.

$ cd /your/path/to/code/projects/demoproject
$ touch .env
$ echo "source /your/path/to/code/virtualenvs/demoproject/bin/activate" > .env
$ cd ..
$ cd demoproject
(demoproject)$

Virtual environment should now have automatically activated. Note that with autoenv you still have to manually deactivate. If you are changing the project you are working on, the second one can be activated without first deactivating the first one.

Install Django and create Django project

After you have created the virtual environment for you project, it is time to install Django and create the Django project. It's important to first create and activate the virtual environment, so your Django will be installed in the virtual environment.

pip

From Python 3.4 onwards virtual environments created with pyvenv have pip installed by default. Use pip to install 3rd party Python packages like Django. See Django installation in official page.

(demoproject)$ pip install Django

The Django files should have appeared in your virtual environment directory.

|-code
   |
   |-projects
   |     |-facetagram
   |     |-instatube
   |
   |-virtualenvs
         |-facetagram
         |-instatube
         |-demoproject
               |-bin
               |-include
               |-pyvenv.cfg
               |-lib
                  |-python3.4
                       |-site-packages
                              |-pip
                              |-django

After you install new libraries with pip, you should run

(demoproject)$ pip freeze > requirements.txt

This updates a file requirements.txt, which contains list of all installed Python (in current virtual environment) libraries. This file can be used to install all missing libraries with a single command. Store this file in the version control.

Install all dependencies listed in the requirements-file with command:

(demoproject)$ pip install -r requirements.txt

Create the Django project

Follow e.g. Django tutorial.

$ cd /your/path/to/code/projects
$ source ./../virtualenvs/demoproject/bin/activate
(demoproject)$ django-admin startproject demoproject
(demoproject)$ cd demoproject
(demoproject)$ python manage.py startapp demoapp

What I usually do in my Django applications is to combine "project" and "app" folders when I know I won't be developing any reusable Django app and that the application will remain relatively small in size. This makes the structure more clear since I don't have to come up with forced separate names for a project and an app.

Settings.py

Some settings are different for your development and production environment. For example, if your application uses a database, it should be separate for development and production. The correct settings should be chosen automatically. Otherwise you are guaranteed to mess things up at some point. Because settings.py is a normal Python file, you can include some logic in it. The simplest way is to find an existing environmental variable, that is always different in your production machine compared to development machines. This could be e.g. LOGNAME.

Now you could build settings.py like the following:

import os

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

PRODUCTION_LOGNAME = 'demouser'  
if os.environ['LOGNAME'] == PRODUCTION_LOGNAME:  
    PRODUCTION = True
else:  
    PRODUCTION = False
DEBUG = not PRODUCTION

# ...

# Postgre in production, sqlite in development.
if PRODUCTION:  
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': 'demoproject_db',
            'USER': 'demoproject_dbuser',
            'PASSWORD': 'demopassword123',
        }
    }
else:  
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
        }
    }

# ...

A bit more enterprisy and recommended way to handle settings is to have one base settings file which contains settings that are common for all the environments for your project. In addition to this base settings file, you then have multiple environment specific settings files, which override or add to the base file.

Folder stucture for settings could be something like this:

|-demoproject
    |
    |-settings
    |    |-base.py
    |    |-development.py
    |    |-production.py
    |    |-staging.py
    |
    |-wsgi.py  

when the automatically generated default was this:

|-demoproject
    |
    |-settings.py
    |-wsgi.py
    |-...

Environment specific settings can override the base setting file, when they begin with from ..settings.base import *

For example, base.py. Note that we added one extra os.path.dirname to BASE_DIRsince this base.pynow lives one folder deeper that the default settings.py.

import os

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False

ALLOWED_HOSTS = []

INSTALLED_APPS = [  
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'myapp',
]

MIDDLEWARE = [  
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'epassikuva.urls'

TEMPLATES = [  
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'epassikuva.wsgi.application'

AUTH_PASSWORD_VALIDATORS = [  
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

LANGUAGE_CODE = 'fi'  
TIME_ZONE = 'UTC'  
USE_I18N = True  
USE_L10N = True  
USE_TZ = True  
STATIC_URL = '/static/'  

and development.py file with some additions and overrides. E.g. here we disable the auth password validation rules for development environment.

from ..settings.base import *

SECRET_KEY = 'secret123'

DEBUG = True

DATABASES = {  
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

AUTH_PASSWORD_VALIDATORS = []  

Django framework knows which settings file to use by looking at the DJANGO_SETTINGS_MODULE environment variable. In the example case, the environment variable should be set to DJANGO_SETTINGS_MODULE="demoproject.settings.development" in development environment.

Why by default Django knows to load the settings.py as the settings file? The automatically generated wsgi.py and manage.py files contain this line: os.environ.setdefault("DJANGO_SETTINGS_MODULE", "demoproject.settings").

I recommend that you modify these lines to point to your new development.pysettings file, i.e. replace os.environ.setdefault("DJANGO_SETTINGS_MODULE", "demoproject.settings") with os.environ.setdefault("DJANGO_SETTINGS_MODULE", "demoproject.settings.development") for both wsgi.py and manage.py.

In your production environment you must then create and set the DJANGO_SETTINGS_MODULE env variable (e.g. in Linux command line, not in Python code like above). It is also recommended to save all passwords and other environment dependent config values to OS env variables and read them from there. See 12-factor app.

Front-end libraries

Your application probably wont be using just server side Python libraries (managed with pip). A webapp might be using front-end libraries like Bootstrap for style and jQuery for small front-end JavaScript tasks.

I use Bower to handle front-end dependencies. Installing Bower requires npm, which comes with Node.js. Install Node.js.

Install Bower

$ npm install -g bower

Initialize bower.json for your project

$ cd /your/path/to/code/projects/demoproject
$ bower init

This runs a short wizard. Afterwards you can modify the created bower.json by hand. bower.json will be the place where front-end dependecies will be saved when you install them. Afterwards you can use the bower.json to automatically install the same libraries in e.g. production server.

Install new front-end libraries

$ bower install jquery --save
$ bower install bootstrap --save

Now your bower.json should look something like this

{
  "name": "demoproject",
  "private": true,
  "ignore": [
    "**/.*",
    "node_modules",
    "bower_components",
    "test",
    "tests"
  ],
  "dependencies": {
    "bootstrap": "~3.3.5",
    "jquery": "~2.1.4"
  }
}

Your folder structure should be like this

|-code
   |
   |-projects
   |     |-facetagram
   |     |-instatube
         |-demoproject
               |-demoapp
               |-demoproject
               |-bower_components
               |      |-bootstrap
               |      |-jquery
               |-.env
               |-bower.json
               |-manage.py
   |
   |-virtualenvs
         |-facetagram
         |-instatube
         |-demoproject

Reinstall all Bower dependencies to match bower.json with

$ bower update

Version control with Git

You should use version control. Git is ok.

What files should be in version control?

If there is no reason to do otherwise, virtual environment files and 3rd party libraries should not be in version control. They can and should be created or fetched with other means. Doing so reduces greatly the amount of data you have in your version control.

If you use a lot of 3rd party packages with your project, it might be tempting to put these packages in version control, so you don't have to install them all separately in all the machines, like production server. Luckily, pip and Bower both can create a simple file listing all used dependencies in your project. This list file should live in your version control, and it should be used to install/update the libraries of your server when you deploy a new version of your application to production.

Database migration scripts should live in version control. Database, like sqlite3 database file, should not be in version control. Just the database connection configuration in settings.py should be in version control.

There are two places where the files that Git ignores can be set. Global and repository specific. Add .gitignore file to your project base dir to add project specific ignored files and directories. Good template for your own .gitignore. Project specific .gitignore should live in version control.

Database migrations

When your app uses a database, it's not just the Python code that evolves, when you develop the app. Also database schema changes with your Django models. Current Django versions have a great built in support for database migrations. Learn how to use them. Database migration scripts allow you to change database schema without losing data.

In short, you need two commands for migrations. Namely

(demoproject)$ python manage.py makemigrations

and

(demoproject)$ python manage.py migrate
  1. Use makemigrations to automatically create the migration script files based on the changes you have made on your Django models.
  2. Use migrate to apply all not-yet-applied migrations to the database.

Normal migration workflow:

  1. Do some changes to your models.
  2. Create corresponding migration scripts with makemigrations.
  3. Run migration scripts (migrate) on your development machine. Check that everything works.
  4. Commit model changes and migration scripts to version control in the same commit.
  5. (Other developers pull the commit and run migration scripts (migrate).)
  6. Pull the commit in production environment. Run migration scripts (migrate).

Deployment with Fabric

It is recommended to get the deployment process to be as easy as possible, as soon as possible. It shouldn't be a hard and feared task to deploy a new version to production! Automated deployment means anyone can deploy anytime a new feature or bug fix has been completed. Automation makes deployment much faster and less error prone. Automating deployment pays back the setup time cost even in the smallest hobby projects.

It is easy to automate Django deployment. With Fabric you basically just make notes of your deployment steps in a certain syntax.

In a fabfile.py you can have separate steps for preparing deployment and doing the actual deployment. I usually have just the deployment step. Fabric can run commands on remote machines. That means you can run deployment command on your local machine to deploy new version on a remote production server.

My typical fabfile.py might look something like this (Gunicorn is the WSGI webserver I'm using to serve the Django project):

from fabric.api import run, env, task  
from fabric.context_managers import cd, prefix

USER = 'username'  
HOST = '111.111.111.111'  # server SSH address  
APP_NAME = 'demoproject'

env.hosts = ['{}@{}'.format(USER, HOST)]

PROJECT_BASE = '/home/{}/{}/'.format(USER, APP_NAME)  
SRC_BASE = PROJECT_BASE  
VENV_BASE = '{}{}env/'.format(PROJECT_BASE, APP_NAME)

def run_venv(command):  
    with prefix('source {}bin/activate'.format(VENV_BASE)):
        run(command)

def install_python_dependencies():  
    with cd(SRC_BASE):
        run_venv("pip install -r requirements.txt")

@task
def gunicorn_stop():  
    run("sudo systemctl stop gunicorn-demoproject")

@task
def gunicorn_start():  
    run("sudo systemctl start gunicorn-demoproject")

@task
def gunicorn_restart():  
    run("sudo systemctl restart gunicorn-demoproject")

def collect_static():  
    with cd(SRC_BASE):
        run_venv(
            'python manage.py collectstatic --noinput --clear')

def db_migrate():  
    with cd(SRC_BASE):
        run_venv("python manage.py migrate")

def pull_new_master():  
    with cd(SRC_BASE):
        run_venv("git checkout master")
        run_venv("git pull origin master")

@task(default=True)
def deploy():  
    """
    Deploy demoproject on production server.
    """
    gunicorn_stop()
    pull_new_master()
    install_python_dependencies()
    db_migrate()
    collect_static()
    gunicorn_start()

This fabfile.py is in my Django project's base dir. Deployment is done like this:

$ fab deploy

When on the project base dir on development machine. This command deploys the latest versions from master branch.

Note that you must have Fabric installed on your development machine outside the project virtual environment if your project uses Python 3, since Fabric only supports Python 2.

You don't need to install Fabric at all on the production server. Fabric will do it's magic over SSH from your local computer.

comments powered by Disqus