People strive for the path of least resistance, and when they do, steps get missed and things go wrong. I've found that if you want a task completed in a specific way, you must make it easier and more beneficial to do it that way than any other way. Luckily we have tools to make the our tasks easy to complete.
Read MoreFabric
Making a multi-branch test server
Problem: How can you asyncronously manage the develop-test-deploy process? If you only have one test server, the testing process will hold up the developers.
A Solution: Allow your test server to host multiple versions of your code. The basic idea is that branch "feature1" becomes live at "http://feature1.test.example.com/".
Our development workflow allows us to publish a branch of our code on our test server so someone else may test it. This allows our team to work on several parallel development and testing paths at once. It is easy easy to manage and doesn't require any additional resources.
Table of Contents:
Prerequisites
There are a few things you need before we get to the code:
A server used for testing your code. You don't need to have a full duplicate setup of your production servers. For example, our test server hosts all the various processes that are on different servers in production, except the database. All our test instances use the same test database on our database server (unless the test instance needs its own database for some reason).
A wildcard DNS record. This record should resolve all subdomains of your test server to the test server IP address. This way you don't have to set up a DNS record for every test instance you create.
Secure shell (ssh) access to the test server. This is kind of a given.
Configuration templates in your code repository. Your test instances require some basic configuration. This example uses three templates: one for the Django settings, one for the Nginx site, and one for Upstart to manage the gunicorn process. We'll cover the templates in more depth later.
Your settings configured in a directory. This example assumes that you have a settings directory with the following files:
__init__.py
base.py
dev.py
production.py
test.py.template
You can do it without this, but you will have to alter the code accordingly.
A bootstrap file for setting up the virtualenv. Our bootstrap script was initially generated from this gist. Basically it sets up a python virtualenv in a directory named "virtualenv" in the root directory of your code. It then uses pip
to install your requirements.txt
.
A couple of variables set in your Fabric environment. Currently the scripts assume env.site_root
is the absolute path to where to deploy your test instance code and env.repo_url
is the URL to your git
repository.
Configuration templates
Each configuration template is processed to replace the string {branchname}
with the name of the test instance.
test.py.template example
from .base import *
DEBUG = True
TEMPLATE_DEBUG = DEBUG
DATABASES = { # NOTE 1
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'test',
'USER': 'dbuser',
'PASSWORD': 'dbpassword',
'HOST': 'dbserver',
'PORT': '',
}
}
MEDIA_ROOT = "/home/www/media/" # NOTE 2
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'staticmedia') # NOTE 3
STATIC_URL = "/static/"
try:
from local_settings import *
except ImportError:
pass
Notes:
The database for each test instance is the same. This isn't typically an issue. However, we can manually create another database and adjust the
test.py
file to point to the new test database.All the test instances share media, but not static media. Each instance has its own
staticmedia
directory.PROJECT_ROOT
is set insettings/base.py
toos.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
nginx-test.conf.template example
upstream gunicorn-{branchname} {
server unix:///var/run/{branchname}.sock; # NOTE 1
}
server {
listen 80;
server_name {branchname} {branchname}.test.example.com;
access_log /var/log/nginx/{branchname}.access.log; # NOTE 2
location /media/static/ {
alias /var/www/{branchname}/staticmedia/;
}
location /media/ { # NOTE 3
root /home/www;
}
location /static/ { # NOTE 3
alias /var/{branchname}/staticmedia/;
}
location / {
proxy_pass http://gunicorn-{branchname};
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
# NOTE 4
auth_basic "Unauthorized Use Prohibited";
auth_basic_user_file /etc/nginx/passwd;
}
add_header X-Whom {branchname};
location ~ /\. { deny all; }
}
Notes:
All the gunicorn processes exist in
/var/run/
.Each test instance has its own access log in
/var/log/nginx/
./media/static/
and/static/
point to the appropriatestaticmedia
directory, while/media/
points to the shared media directory.We have a basic user/password required to access each test instance.
upstart-test.conf.template example
description "{branchname}"
start on runlevel [2345]
stop on runlevel [06]
respawn
respawn limit 10 5
# NOTE 1
exec /var/www/{branchname}/start_gunicorn.sh \
--settings settings.test --debug --workers=1
Notes:
Since the upstart configuration references it, here is our
start_gunicorn.sh
#!/bin/bash set -e HOMEDIR="$( cd -P "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" CONF="$HOMEDIR/conf/gunicorn_conf.py" NAME="$(basename $HOMEDIR)" cd $HOMEDIR source virtualenv/bin/activate exec $HOMEDIR/virtualenv/bin/python manage.py run_gunicorn --config $CONF $@
The
$@
at the end of the last line passes on all arguments. This allows us to specify the settings to use from the upstart script and use the same command for both test and production servers.
The Fabric tasks
Fabric tasks automate the whole process. I've put the whole file on github as a gist. I'll go over each task specifically. The basic commands are:
make_test_instance
remove_test_instance
stop_test_instance
start_test_instance
list_test_instances
update_test_instance
A typical workflow looks like:
- Make a new feature branch locally.
- Write lots of good code.
- Push your branch to the origin server.
- Execute
fab make_test_instance:branchname
- Test.
- Make changes locally, and push your code to origin.
- Execute
fab update_test_instance:branchname
- Test more.
- Merge your finished code into master.
- Execute
fab remove_test_instance:branchname
The nice part is that the developer can move on to another branch and continue working while testing progresses, or continue on the same one if possible while awaiting feedback on the code the developer already pushed.
make_test_instance
This task is designed to be idempotent, meaning you can run it over and over without negative effects. If there is an error the first time you run it, running it again will simply continue where you left off (assuming you've fixed whatever caused the original error).
Example Usage:
fab -H user@test.example.com make_test_instance:featurebranch
fab -H user@test.example.com make_test_instance:featurebranch,test
make_test_instance
has one required argument, branchname
. It is the name of the branch which it will check out. The second, optional argument, instance_name
allows you to give it a different name. Without instance_name
, the test instance is named branchname
.
Code:
@task
def make_test_instance(branchname, instance_name=""):
"""
Make a stand-alone instance using branch <branchname>
Named using <instance_name> or <branchname>
"""
if not instance_name:
instance_name = branchname
instance_dir = env.site_root + instance_name
if not exists(instance_dir):
with cd(env.site_root):
run('git clone %s %s' % (env.repo_url, instance_name))
with cd(instance_dir):
run('git checkout %s' % branchname)
else:
with cd(instance_dir):
run("git pull")
_process_template((instance_dir, 'settings', 'test.py.template'),
instance_name)
run('mkdir -p %s' % os.path.join(instance_dir, 'staticmedia', 'CACHE'))
sudo('chmod -R a+w %s' % os.path.join(instance_dir, 'staticmedia', 'CACHE'))
_bootstrap(instance_name, 'test')
sudo('chgrp -R www-data %s%s/staticmedia' % (env.site_root, instance_name))
sudo('chmod -R g+w %s%s/staticmedia' % (env.site_root, instance_name))
upstart_conf = _process_template((instance_dir, 'conf', 'upstart-test.conf.template'),
instance_name)
upstart_link = "/etc/init/%s.conf" % instance_name
sudo(_make_link_cmd(upstart_conf, upstart_link))
sudo('initctl reload-configuration')
web_conf = _process_template((instance_dir, 'conf', 'nginx-test.conf.template'),
instance_name)
web_link = '/etc/nginx/sites-available/%s' % instance_name
if not exists(web_link):
sudo(_make_link_cmd(web_conf, web_link))
sudo('nxensite %s' % instance_name)
sudo('/etc/init.d/nginx reload')
Depending on your server configurations and setups you might need to dramatically change this task.
First it either clones the repo into the appropriate directory and checks out the specified branch, or pulls the latest version of the branch.
Then it creates the test settings by processing the test settings template.
Since the static media is separate for each test instance, it creates the directory which will contain the static media, and the CACHE
directory that django-compressor
uses to store its files.
The bootstrapping process basically executes the bootstrap.py
file mentioned above, and then executes ./manage.py collectstatic
, ./manage.py syncdb
and ./manage.py migrate
to get things set up. (See the linked lines to see the specifics.)
After bootstrapping the instance, it alters the permissions on the static media directory, and creates an upstart configuration using the included upstart template. It creates a symbolic link to this new file from the /etc/init/
directory and tells upstart to reload its configurations. This part allows us to start and stop test instances with start feature1
and stop feature1
commands, respectively.
Finally we process the web configuration template and create a link to it from the /etc/init.d/nginx/sites-available/
directory. We can tell nginx to enable the site and reload its configuration to get things going.
remove_test_instance
This task simply undoes everything done in make_test_instance
. When you are done with an instance, this cleans up everything nicely.
Example Usage:
fab -H user@test.example.com remove_test_instance:instance_name
remove_test_instance
requires only one argument: the name of the instance to remove.
Code:
@task
def remove_test_instance(instance_name):
"""
Remove a test instance and remove all support scripts and configs
"""
nginx_name = '/etc/nginx/sites-enabled/%s' % instance_name
if exists(nginx_name):
sudo('nxdissite %s' % instance_name)
sudo('/etc/init.d/nginx reload')
nginx_name = '/etc/nginx/sites-available/%s' % instance_name
if exists(nginx_name):
sudo('rm %s' % nginx_name)
upstart_link = "/etc/init/%s.conf" % instance_name
if exists(upstart_link):
sudo('stop %s' % instance_name)
sudo('rm %s' % upstart_link)
sudo('initctl reload-configuration')
instance_dir = env.site_root + instance_name
if exists(instance_dir):
sudo('rm -Rf %s' % instance_dir)
First it unloads and unlinks the site nginx site configuration. Then it does the same thing for upstart. Finally it removes everything in the instance directory.
stop_test_instance
, start_test_instance
, restart_test_instance
Each of these commands allows you to stop, start or restart a specific test instance, or every test instance if you don't specify an instance name.
Example Usage:
fab -H user@test.example.com stop_test_instance:feature1
fab -H user@test.example.com restart_test_instance
Code:
def _instance_mgmt(cmd, instance_name=None):
"""
Executes <cmd> against <instance_name> or every instance
"""
env.warn_only = True
if instance_name is not None:
instances = [instance_name]
else:
instances = _list_test_instances()
for item in instances:
sudo("%s %s" % (cmd, item.strip()))
Each of the tasks calls the same underlying function with just a different command. If it doesn't have an instance name, it gets a list of all the instances and executes the command on each of them.
list_test_instances
Sometimes you need a quick update as to what test instances you have out there and their status. This command prints out a nice listing of each and their status.
Example Usage:
fab -H user@test.example.com list_test_instances
Code:
@task
def list_test_instances():
"""
List all the test instances on the test server
"""
instances = _list_test_instances()
output = ["", "Instance Name Status", "-------------- -------------"]
with hide('running', 'stdout'):
for instance in instances:
line = run('status %s' % instance)
line = line.replace(",", "")
line = line.split(" ")
if len(line) == 2:
output.append(red(line[0].ljust(15)) + line[1])
elif len(line) == 4:
output.append(green(line[0].ljust(15)) + line[1])
print "\n".join(output)
This code gets the listing of all the instances, and then queries the test server for the status of each of them. Then it formats the list with stopped instances in red and running intances in green.
update_test_instance
After you've updated a branch, deploying your changes to the test instance is as easy as running this command.
Example Usage:
fab -H user@test.example.com update_test_instance:feature1
update_test_instance
requires the name of the test instance to update.
Code:
@task
def update_test_instance(instance_name):
"""
Update the <instance_name>
"""
instance_dir = env.site_root + instance
settings = "settings.test"
with cd(instance_dir):
run("git pull")
with prefix('source virtualenv/bin/activate'):
run("pip install -r requirements.txt")
run("./manage.py collectstatic --noinput --verbosity 0 --settings %s" % settings)
run("./manage.py syncdb --settings %s" % settings)
run("./manage.py migrate --delete-ghost-migrations --settings %s" % settings)
sudo("restart %s" % instance_name)
This command simply pulls the latest code, installs the requirements (if nothing has changed in requirements.txt, this does nothing), collects the static media, updates the database and finally restarts the test instance.