David Winterbottom (@codeinthehole)

PyPI README badges

Thanks to @kuramanga, it's now possible to add shiny PyPi badges to your Python project READMEs that indicate the latest released version on PyPI and the total number of downloads.

This screenshot is taken from django-oscar's README.

Embed these badges in your own repo as Restructured text:

.. image:: https://pypip.in/v/$REPO/badge.png
    :target: https://crate.io/packages/$REPO/
    :alt: Latest PyPI version

.. image:: https://pypip.in/d/$REPO/badge.png
    :target: https://crate.io/packages/$REPO/
    :alt: Number of PyPI downloads

or Markdown:

[![PyPi version](https://pypip.in/v/$REPO/badge.png)](https://crate.io/packages/$REPO/)
[![PyPi downloads](https://pypip.in/d/$REPO/badge.png)](https://crate.io/packages/$REPO/)

The code is available ...

A useful Git post-checkout hook for Python repos

Every now and again, an innocent python developer checks out a new Git branch then proceeds to bang their head against a bug caused by an orphaned .pyc file from the previous branch. Since *.pyc files are typically in the repo's .gitignore file, they are not removed when switching branches and can cause issues if the corresponding .py is removed.

This can be neatly addressed through a 'post checkout' hook which deletes all such files. Here is such a script, which also removes empty folders and prints a summary:

#!/usr/bin/env bash

# Delete .pyc files and empty directories from root of project
cd ./$(git rev-parse --show-cdup)

# Clean-up
find . -name ".DS_Store" -delete

NUM_PYC_FILES=$( find . -name "*.pyc" | wc -l | tr ...

Disable database access when writing unit tests in Django

Consider this curio:

import mock
from django.utils.functional import curry

no_database = curry(
    mock.patch, 'django.db.backends.util.CursorWrapper',
    Mock(side_effect=RuntimeError("Using the database is not permitted")))

This snippet creates a decorator that can wrap a test case or method and raises an exception if the database is accessed. This can be useful if you're a puritan about true unit tests.

Use by wrapping a TestCase subclass:

from django.test import TestCase

@no_database()
class UnitTestCase(TestCase):
    ...

or method:

from django.test import TestCase

class UnitTestCase(TestCase):

    @no_database()
    def test_something(self):
        ...

This snippet is a reformulation of one from Carl Meyer's excellent 'Testing and Django' (about 24 minutes in).

Challenge: create a similar decorator that prevents all ...

How to install PIL on 64-bit Ubuntu 12.04

Problem

You want to install PIL on 64-bit Ubuntu 12.04 (Precise Pangolin).

Solution

With pip already installed, install the required development packages:

$ sudo apt-get install python-dev libjpeg-dev libfreetype6-dev zlib1g-dev

and symlink the three image libraries into /usr/lib:

$ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libfreetype.so /usr/lib/
$ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libjpeg.so /usr/lib/
$ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libz.so /usr/lib/

PIL should now install with support for JPEGs, PNGs and FreeType, as indicated by the compilation output:

--------------------------------------------------------------------
PIL 1.1.7 SETUP SUMMARY
--------------------------------------------------------------------
version       1.1.7
platform      linux2 2.7.3 (default, Apr 20 2012, 22:39:59)
              [GCC 4.6.3]
--------------------------------------------------------------------
*** TKINTER support ...

Converting Github issues into pull requests

Using the Hub library, it's possible to convert Github issues into pull requests. This gives rise to a useful Github workflow which this article describes.

This is nothing new; it's been written about before. However, this is something I do all the time whilst developing Oscar and I'm fed up with explaining it. This article is a reference I can point people at.

Workflow

Discuss

Discuss an idea for a new feature on the project mailing list. Agree on what needs to be done.

Specify

Create a Github issue for the feature.

It's often useful to write the ticket as a brief functional spec, documenting the requirements as user stories. Github's support for checkboxes in ...

Altering Postgres table columns with South

Problem

You're using Postgres with Django.

You change a field type of one of your models, generate an --auto South migration and attempt to run it. However, South chokes on the new migration complaining that the data in the column cannot be cast to the new type.

For instance, I recently changed a CharField to a TimeField but the corresponding migration lead to:

Running migrations for stores:
- Migrating forwards to 0009_auto__chg_field_openingperiod_start__chg_field_openingperiod_end.
> stores:0008_auto__del_unique_store_slug
> stores:0009_auto__chg_field_openingperiod_start__chg_field_openingperiod_end
FATAL ERROR - The following SQL query failed: ALTER TABLE "stores_openingperiod" ALTER COLUMN "start" TYPE time, ALTER COLUMN "start" DROP NOT NULL, ALTER COLUMN "start" DROP DEFAULT;
The error was: column "start" cannot be cast to type time without time zone

Postgres did not know ...

Configuring logging for Postgres.app

Problem

You're using Postgres.app on a Mac for local development but are getting SQL errors from your application. You're seeing an error message:

ERROR:  current transaction is aborted, commands ignored until end of
transaction block

This isn't very useful: you want to know which query is generating the error.

Solution

Turn on Postgres' logging and watch the log files when the error is generated.

This is done by editing ~/Library/Application Support/Postgres/var/postgresql.conf and setting:

logging_collector = on
log_directory = 'pg_log'

then restarting Postgres to pick up the new settings. You can then watch the log files to find out which queries are failing:

$ tail -f ~/Library/Application\ Support/Postgres/var/pg_log/*

Discussion

By default ...

Mathematics and engineering

Structure and Interpretation of Computer Programs is full of interesting footnotes. Here's a good one about the Fermat test for prime numbers:

Numbers that fool the Fermat test are called Carmichael numbers, and little is known about them other than that they are extremely rare. There 255 Carmichael below 100,000,000. The smallest few are 561, 1105, 1729, 2465, 2821 and 6601. In testing primality of very large numbers chosen at random, the chance of stumbling upon a value that fools the Fermat test is less than the chance that cosmic radiation will cause the computer to make an error in carrying out a correct algorithm. Considering an algorithm to be inadequate for the first reason but not ...

Effective pull requests and other good practices for teams using github

I work at an agency where we pay $200 a month to Github for their platinum plan. This article is a summary of an internal talk I gave on making the most of our subscription.

There's nothing original here: it's just a collection of tips that I've harvested over the last few years. I'm publishing this article mainly so I have something to refer future employees to.

Use pull requests

Pull requests are an excellent tool for fostering code review. If you're using Github for team projects, you should be using these extensively.

Many people don't realise that you can make pull requests between two branches of the same repository (the so-called "shared repository ...

How to chroot a user in Ubuntu 12.04

External parties often need to upload data to your application. Sadly, most ask for a FTP server. Push back against this and suggest they use sFTP.

This article explains how to set-up a chroot-ed user in Ubuntu 12.04 so that an external party can upload data to your application securely.

This is mainly for my own reference.

User set-up

Create user with a dummy shell:

$ adduser --shell=/bin/false barry

and alter the ownership and permissions of their home folder:

$ chown root:barry /home/barry
$ chmod 755 /home/barry

Now create a folder to upload to:

$ mkdir /home/barry/uploads
$ chown barry:barry /home/barry/uploads
$ chmod 755 /home/barry/uploads

SSH config

Edit /etc/ssh/sshd_config and comment ...

Prefer WebTest to Django's test client for functional tests

Since watching Carl Meyer's superb 'Testing and Django' talk, I've been using Ian Bicking's WebTest library for functional tests, via django-webtest. I've been really impressed and I'd like to stress one of Carl's points - that using WebTest for functional tests is superior to using the Django client.

Why?

Several reasons - here's a few:

WebTest allows you to model a user's experience much more closely as it is smart about mark-up. Instead of hand-crafting GET and POST requests, you can use the WebTest API to follow links and submit forms - this is what users actually do. As a result, your tests accurately capture user stories.
A corollary to the last point is that ...

Cacheback - asynchronous cache refreshing for Django

Inspired by Jacob Kaplan-Moss's excellent talk "Django doesn't scale" at this year's OSCon, I've put together a Django package for re-populating caches asynchronously.

It provides a simple API for wrapping expensive read operations that caches results and uses Celery to repopulate items when they become stale. It can be used as a decorator for simple cases but provides an extensible class for more fine-grained control. It also provides helper classes for working with querysets.

The package is MIT-licensed, published to PyPI and the source is available on Github. It's best explained with an ...

Example

Consider a view that renders a user's tweets:

from django.shortcuts import render
from myproject.twitter import fetch_tweets

def show_tweets ...

Use models for uploads

All Django developers will deal with file uploads at some point. I contend that it's a good practice to use models to capture the upload metadata and to track processing status. This article explains how and why.

An e-commerce example

Suppose your e-commerce application allows admins to upload CSV files to update product stock levels (a common requirement). A typical file may comprise a SKU and a stock level:

9781231231999,0
9781231231999,4
9781231231999,2
...

Django's docs detail a common pattern for dealing with file uploads such as this. The steps are generally:

Validate the form submission;
Write upload data to permanent storage;
Process the file;
Delete the file (optional)

For example:

def handle_upload(request):
    if request.method ...

Vim macros for adding i18n support to Django templates

Problem

You want to add i18n support to an existing project. One part of this is modifying all templates to use the {% trans "..." %} block around all hard-coded strings.

When you have a lot of templates, this gets pretty tedious.

Solution

Use Vim macros!

Macro 1 - Convert tag text

To convert

<h1>Welcome to my site</h1>

<h1>{% trans "Welcome to my site" %}</h1>

use the macro

vitc{% trans "" %}<ESC>4hp

which breaks down as:

vit - select content inside the tag;
c{% trans "" %} - change tag content to be {% trans "" %} while saving the original tag content to the anonymous register;
<ESC>4hp - move the cursor to the first speech mark and paste the original tag contents. Note that <ESC> is one ...

A data migration for every Django project

How to use a South data migration to avoid accidentally sending emails from example.com.

Problem

Consider the following snippet from Django's docs [1] for sending a confirmation email:

from django.contrib.sites.models import Site
from django.core.mail import send_mail

def register_for_newsletter(request):
    current_site = Site.objects.get_current()
    send_mail(
        'Thanks for subscribing to %s alerts' % current_site.name,
        'Thanks for your subscription. We appreciate it.\n\n-The %s team.' % current_site.name,
        'editor@%s' % current_site.domain,
        [user.email]
    )

Here the domain for the email sender is taken from the 'current site' instance, which is controlled by Django's 'Sites' framework and accessible by a custom method on the manager of the Site model.

By default, a Site instance is ...