Steve Losh

Writing Vim Plugins

2011-09-06T09:13:00Z

A while ago I wrote a post about switching back to Vim. Since then I’ve written two plugins for Vim, one of which has been officially “released”.

A couple of people have asked me if I’d write a guide to creating Vim plugins. I don’t feel confident enough to write an official “guide”, but I do have some advice for Vim plugin authors that might be useful.

Other People Who Know More Than I Do

Writing two decently-sized Vim plugins has given me some experience, but there are a lot of people that know far more than I do. There are two in particular that come to mind. I’d love for them to write some guides (or even books) about modern-day Vim scripting.

Tim Pope

The first is Tim Pope. He’s written a ton of Vim plugins like Pathogen, Surround, Repeat, Speeddating and Fugitive. Each of those is clear, focused and polished.

It would be awesome to read a guide on the ins and outs of Vim scripting by him.

Scrooloose

The other person that comes to mind is Scrooloose, author of NERDTree, NERDCommenter and Syntastic.

His plugins are large and full-featured but work incredibly well, considering how tricky and painful Vimscript is to work with. I’d love to read a guide on writing large-scale Vim plugins by him.

Be Pathogen-Compatible

It’s 2011. When writing your plugin, please make its source compatible with Pathogen. It’s very easy to do this — just set up your project’s files like this:

yourplugin/
    doc/
        yourplugin.txt
    plugin/
        yourplugin.vim
    ...
    README
    LICENSE

This will let users use Pathogen (or Vundle) to install and use your plugin.

The days of “unzip and drag the files into the right directories” and the horror of Vimballs are over. Pathogen and Vundle are the right way to manage plugins, so let your users use them.

Please, For the Love of God, Use normal!

My first piece of actual scripting advice is something simple but important. If you’re writing a Vim plugin and need to perform some actions, you might be tempted to use normal. Don’t. Instead, you need to use normal!.

normal! is like normal, but ignores mappings the user has set up. If you use plain old normal dd and I’ve remapped dd to do something else, the call will use my mapping and probably not do what your plugin expects. Using normal! ensures that the call will do what you expect no matter what the user has mapped.

This is a single instance of a more general theme. Vim is very customizable and users will do lots of crazy things in their .vimrc files. If a key can be mapped or a setting changed, you have to assume that some user of your plugin will have mapped or changed it.

Mapping Keys the Right Way

Most plugins add key mappings to make them easier to use. Unfortunately this can be tricky to get right. You can never tell what keys your users have already mapped themselves, and shadowing someone’s favorite key mapping will break their muscle memory and annoy them to no end.

When to Map Keys

The first question to ask is whether your plugin needs to map keys itself at all.

My Gundo plugin has only one feature that needs to be mapped to a key in order to make it useful: the “toggle Gundo” action.

Gundo doesn’t map this key itself, because no matter what “default” mapping I pick someone will have already mapped it. Instead I added a section right in the README file that shows how a user can map the key themselves:

nnoremap <F5> :GundoToggle<CR>

By making users add this line to their .vimrc themselves it shows them which key is used to toggle Gundo (which they would have to know anyway) and also makes it obvious how to change it to suit their taste.

imap and nmap are Pure Evil

Sometimes forcing the user to map their own keys won’t work. Perhaps your plugin has many mappings that would be tedious for a user to set up manually (like my Threesome plugin), or its mappings are mnemonic and wouldn’t really make sense if mapped to other keys.

I’ll talk more about how to deal with this in a moment, but the most important thing to remember when mapping your own keys is that you must always, always, always use the noremap forms of the various map commands.

If you map a key with nmap and the user has remapped a key that your mapping uses, your mapped key will almost certainly not do what you want. Using nnoremap will ignore user mappings and do what you expect.

This is the same principle as normal and normal!: never trust your users’ configurations.

Let Me Configure Mappings

If you feel that your plugin must map some keys, please make those mappings configurable in some way.

There are a number of ways to do this. The easiest way is to provide a configuration option that disables all mappings. The user can them remap the keys as they see fit. For example:

if !exists('g:yourplugin_map_keys')
    let g:yourplugin_map_keys = 1
endif

if g:yourplugin_map_keys
    nnoremap <leader>d :call <sid>YourPluginDelete()<CR>
endif

Normal users will get the mappings automatically set up for them, and power users can remap the keys to whatever they wish to avoid shadowing their own mappings.

If your plugin’s mappings all start with a common prefix (like <leader> or <localleader>) you have another option: allow users to configure this prefix. This is the approach I’ve used in Threesome. It works like this:

if !exists('g:yourplugin_map_prefix')
    let g:yourplugin_map_prefix = '<leader>'
endif

execute "nnoremap"  g:yourplugin_map_prefix."d"  ":call <sid>YourPluginDelete()<CR>"

The execute command lets you build the mapping string dynamically so your users can change the mapping prefix.

There is a third option for solving this problem: the hasmapto() Vim function. Some plugins will use this to map a command to a key unless the user has already mapped that command to something else. I don’t personally like this option because it feels less clear to me, but I know other people feel differently so I wanted to mention it.

Localize Mappings and Settings

The next step in being a good Vim plugin author is to try to minimize the effects of your key mappings and setting changes. Some plugins will need to have global effects but others will not.

For example: if you’re writing a plugin for working with Python files it should only take effect for Python buffers, not all buffers.

Localizing Mappings

Key binding are easy to localize to single buffers. All of the noremap commands can take an extra <buffer> argument that will localize the mapping to the current buffer.

" Remaps <leader>z globally
nnoremap <leader>z :YourPluginFoo<cr>

" Remaps <leader>z only in the current buffer
nnoremap <buffer> <leader>z :YourPluginFoo<cr>

However, the problem is that you need to run this command in every buffer you want the mapping active. To do this your plugin can use an autocommand. Here’s a full example, using this concept plus the previously mentioned configuration options:

if !exists('g:yourplugin_map_keys')
    let g:yourplugin_map_keys = 1
endif

if !exists('g:yourplugin_map_prefix')
    let g:yourplugin_map_prefix = '<leader>'
endif

if g:yourplugin_map_keys
    execute "autocommand FileType python" "nnoremap <buffer>" g:yourplugin_map_prefix."d"  ":call <sid>YourPluginDelete()<CR>"
endif

Now your plugin will define a key mapping only for Python buffers, and your users can disable or customize this mapping as they see fit.

This mapping command is quite ugly. Unfortunately that’s the price of using Vimscript and trying to make a plugin that will work for many users. Later I’ll talk about one possible solution to this ugliness.

Localizing Settings

Just as you should make mappings local to buffers when appropriate, you should do the same with settings like foldmethod, foldmarker and shiftwidth. Not all settings can be set locally in a buffer. You can read :help <settingname> to see if it’s possible.

You can use setlocal instead of set to localize settings to individual buffers. Like with mappings you’ll need to use an autocommand to run the setlocal command every time the users opens a new buffer.

Autoload is Your Friend

If your plugin is something that users will be using all the time you can skip this section.

If you’re writing something that will only be used in specific cases, you can help your users by using Vim’s autoload functionality to delay loading its code until the user actually tries to use it.

The way autoload works is fairly simple. Normally you would bind a key to call one of your plugin’s functions with something like this:

nnoremap <leader>z :call YourPluginFunction()<CR>

You can use autoloading by prepending yourplugin# to the name of the function:

nnoremap <leader>z :call yourplugin#YourPluginFunction()<CR>

When this mapping is run, Vim will do the following:

Check to see if YourPluginFunction is already defined. If so, call it.
Otherwise, look in ~/.vim/autoload/ for a file named yourplugin.vim.
If it exists, parse and load the file (which presumably defines YourPluginFunction somewhere inside of it).
Call the function.

This means that instead of putting all of your plugin’s code in plugin/yourplugin.vim you can put just the key mapping code there and pull the rest out into autoload/yourplugin.vim.

If your plugin has a decent amount of code this can reduce the startup time of Vim by a significant amount.

Check out the full documentation of autoload by running :help autoload to learn much more.

Backwards Compatibility is a Big Deal

Once you’ve written your Vim plugin and released it into the wild, you have to maintain it. Users will find bugs and ask for new features.

Part of being a responsible developer of any kind, including a Vim plugin author, is maintaining backwards compatibility, especially for tools that users will use every day and burn into their muscle memory. Users rely on tools to work, and tools that break backwards compatibility will quickly lose users’ trust.

Maintaining backwards compatibility will cause your plugin’s code to get crufty in spots, but it’s the price of maintaining your users’ happiness.

What Matters for Backards Compatibility?

For a Vim plugin the most important part of staying backwards compatible is ensuring that key mappings, customized or not, continue to do what users expect.

If your plugin maps key X to do Y, then pressing X should always do Y, even if you change how Y is called by renaming Y to Z. This may mean changing Y into a wrapper function which simply calls Z.

There are many other aspects of backwards compatibility that you will have to consider, depending on the purpose of your plugin. The rule of thumb you should follow is: if a user uses this plugin on a daily basis and has its usage burned into their muscle memory, updating the plugin should not make them relearn anything.

Use Semantic Versioning So I Can Stay Sane

A fast, simple, easy way to document your plugin’s state is to use semantic versioning.

Semantic versioning is simply the idea that instead of picking arbitrary version numbers for releases of your project, you use version numbers that describe the backwards-compatible state in a meaningful way.

In a nutshell, these rules describe how you should select version numbers for new releases:

Version numbers have three components: major.minor.bugfix. For example: 1.2.4 or 2.13.0.
Versions with a major version of 0 (e.g. 0.2.3) make no guarantees about backwards compatibility. You are free to break anything you want. It’s only after you release 1.0.0 that you begin making promises.
If a release introduces backwards-incompatible changes, increment the major version number.
If a release is backwards-compatible, but adds new features, increment the minor version number.
If a release simply fixes bugs, refactors code, or improves performance, increment the bugfix version number.

This simple scheme makes it easy for users to tell (in a broad sense) what has changed when they update your project.

If only the bugfix number has changed they can update without fear and continue on without worrying about changes unless they’re curious.

If the minor version number has changed they might want to look at the changelog to see what new features they may want to take advantage of, but if they’re busy they can simply update and move on.

If the major version number has changed it’s a major red flag, and they’ll want to read the changelog carefully to see what is different.

Some people don’t like semantic versioning for the following reason:

If I have to increment the major version number every time I make backwards-incompatible changes, I’ll quickly be at ugly versions like 24.1.2!

To this I say: “Yes, but if that happens you’re doing things wrong in the first place.”

Keep your project in “beta” (i.e. version 0.*.*) for as long as you need to experiment freely. Take your time and make sure you’ve gotten things (mostly) right. Once you release 1.0.0 it’s time to start being responsible and caring about backwards compatibility.

Breaking functionality all the time harms your users by reducing their productivity and frustrating them. Yes, it means adding some cruft to your code over time, but it’s the price of not being evil.

Document Everything

A critical part of releasing a Vim plugin to the world is writing documentation for it. Vim has fantastic documentation itself, so your plugins should follow in its footsteps and provide thorough docs.

Pick Some Requirements and Stick to Them

The most important part of your documentation is telling users what they need to have in order to use your plugin. Vim runs on nearly every system imaginable and can be compiled in many different ways, so being specific about your plugin’s requirements will save users a lot of trial and error.

Does your plugin only work with Vim version X.Y or later?
Does it require Python/Ruby/etc support compiled in? Which version?
Does it not work on Windows?
Does it rely on an external tool?

If the answer to any of those questions is “yes”, you must mention it in the documentation.

Write a README

The first step to documenting your plugin is to write a README file for the repository. You can also use the text of this file as the description if you upload your plugin to the vim website, or the content of your plugin’s website if you create one for it.

Some examples of things to include in your README are:

An overview of what the plugin does.
Screenshots, if possible.
Requirements.
Installation instructions.
Common configuration options that many users will want to know.
Links to:
- A canonical web address to find the plugin.
- The bug tracker for the plugin.
- The source code or repository of the plugin.

Create a Simple Website

This isn’t strictly necessary, but having a simple website for your plugin is an extra touch that makes it seem more polished.

It also gives you a canonical URL that people can visit to get the latest information about your plugin.

I’ve made simple sites for both of my plugins: Gundo and Threesome. Feel free to use them as an example or even take their code and use it for your own plugin sites if you like.

Write a Vim Help Document

The bulk of your plugin’s documentation should be in the form of a Vim help document. Users are used to using Vim’s :help and they’ll expect to be able to use it to learn about your plugin.

Creating a help document is as easy as creating a doc/yourplugin.txt file in your project. It will be indexed automatically by pathogen#helptags() so your users will have the docs at their fingertips.

Two easy ways to learn the syntax of help files are by reading :help help-writing and using an existing plugin’s help file as an example.

Take your time and craft a beautiful help file you can be proud of. Don’t be afraid to add a bit of personality to your docs to break the dryness. The syntastic help file is a great example (especially the About section).

Things to include in your documentation:

A brief overview of the plugin.
A more in-depth description of how the plugin is used.
Every single key mapping the plugin creates.
Ways to extend the plugin, if applicable.
All configuration variables (including their default values!).
The plugin’s changelog.
The plugin’s license.
Links to the plugin’s repository and bug tracker.

In a nutshell: your help file should contain anything a user would ever need to know about your plugin.

Keep a Changelog

The last part of documenting your project is keeping a changelog. You can skip this while your project is still in “beta” (i.e. less than version 1.0.0) but once you officially release a real version you need to keep your users informed about what has changed between releases.

I like to include this log in the README, the plugin’s website, and the documentation to make it as easy as possible for users to see what’s changed.

Try to keep the language of the changelog at a high enough level for your users to understand without knowing anything about the implementation of your plugin. Things like “added feature X” and “fixed bug Y” are great, while things like “refactored the inner workings of utility function Z” are best left in commit messages.

Making Vimscript Palatable

The worst part about writing Vim plugins is, without a doubt, dealing with Vimscript. It’s an esoteric language that’s grown organically over the years seemingly without any strong design direction.

Features are added to Vim, then Vimscript features are added to control those features, then hacky workarounds are added for flexibility.

The syntax is terse, ugly and inconsistent. Is " foo a comment? Sometimes.

Much of the time you’ll spend writing your first plugin will be learning how to do things in Vimscript. The help documentation on all of its features is thorough, but it can be hard to find what you’re looking for if you don’t know the exact name. Looking through other plugins is often very helpful in pointing you toward what you need.

There are a couple of ways to ease the pain of Vimscript, and I’ll briefly talk about two of them here.

Wrap. Everything.

The first piece of advice I have is this: if you want to make your plugins readable and maintainable then you need to wrap up functionality even more than you would in other languages.

For example, my Gundo plugin has a few utility functions that look like this:

function! s:GundoGoToWindowForBufferName(name)"{{{
    if bufwinnr(bufnr(a:name)) != -1
        exe bufwinnr(bufnr(a:name)) . "wincmd w"
        return 1
    else
        return 0
    endif
endfunction"}}}

This function will go to the window for the given buffer name and gracefully handle the case where the buffer/window does not exist. It’s verbose but much more readable than the alternative of using that if statement in every place I need to switch windows.

As you write your plugin you’ll “grow” a number of these utility functions. Any time you duplicate code you should think about creating one, but you should also do so any time you write a particularly hairy line of Vimscript. Pulling complex lines out into named functions will save you a lot of reviewing and rethinking down the line.

Scripting Vim with Other Languages

Another option for making Vimscript less painful is to simply not use it much at all. Vim includes support for creating plugins in a number of other languages like Python and Ruby. Many plugin authors choose to move nearly all of their code into another language, using a small Vimscript “wrapper” to expose it to the user.

I decided to try this approach with Threesome after seeing it used in the vim-orgmode plugin to great effect. Overall I consider it to be a good idea, with a few caveats.

First, using another language will requires your plugin’s users to use a version of Vim compiled with support for that version. In this day and age it’s usually not a problem, but if you want your plugin to run everywhere then it’s not an option.

Using another language adds overhead. You need to not only learn Vimscript but also the interface between Vim and the language. For small plugins this can add more complexity to the project than it saves, but for larger plugins it can pay for itself. It’s up to you to decide whether it’s worth it.

Finally, using another language does not entirely insulate you from the eccentricities of Vimscript. You still need to learn how to do most things in Vimscript — using another language simply lets you wrap most of this up more neatly than you otherwise could.

Unit Testing Will Make You Drink

Unit testing (and other types of testing) is becoming more and more popular today. In particular the Python and Ruby communities seem to be getting more and more excited about it as time goes on.

Unfortunately, unit testing Vim plugins lies somewhere between “painful” and “garden-weaseling your face” on the difficulty scale.

I tried adding some unit tests to Gundo, but even after looking at a number of frameworks I was spending hours simply trying to get my tests to function.

I didn’t even bother trying to add tests to Threesome because for every hour I would have spent fighting Vim to create tests I could have cleaned up the code and fixed bugs instead.

I’ll gladly change my opinion on the subject if someone writes a unit testing framework for Vim that’s as easy to use as Cram. In fact, I’ll even buy the author a $100 bottle of scotch (or whatever they prefer).

Until that happens I personally don’t think it’s worth your time to unit test Vim plugins. Spend your extra hours reading documentation, testing things manually with a variety of settings, and thinking hard about your code instead.

TL;DR

Writing Vim plugins is tricky. Vimscript is a rabbit hole of sadness and despair, and trying to please all your users while maintaining backwards compatibility is a monumental task.

With that said, creating something that people use every day to help them make beautiful software projects is extremely rewarding. Even if your plugin doesn’t get many users, being able to use a tool you wrote is very satisfying.

So if you’ve got an idea for a plugin that would make Vim better just sit down, learn about Vimscript, create it, and release it so we can all benefit.

If you have any questions or comments feel free to hit me up on Twitter. You might also enjoy following @dotvimrc where I try to tweet random, bite-sized lines you might like to put in your .vimrc file.

Django Advice

2011-06-30T08:30:00Z

For the past year and a half or so I’ve been working full-time at Dumbwaiter Design doing Django development. I’ve picked up a bunch of useful tricks along the way that help me work, and I figured I’d share them.

I’m sure there are better ways to do some of the things that I mention. If you know of any feel free to hit me up on Twitter and let me know.

Also: this entry was written over several months, so if there are inconsistencies let me know and I’ll try to fix them.

Vagrant

I used to develop Django sites by running them on my OS X laptop locally and deploying to a Linode VPS. I had a whole section of this post written up about tricks and tips for working with that setup.

Then I found Vagrant.

I just deleted the entire section of this post I wrote.

Vagrant gives you a better way of working. You need to use it.

Why Vagrant?

If you haven’t used it before, Vagrant is basically a tool for managing VirtualBox VMs. It makes it easy to start, pause, and resume VMs. Instead of installing Django in a virtualenv and developing against that, you run a VM which runs your site and develop against that.

This may not sound like much, but it’s kind of a big deal. The critical difference is that you can now develop against the same setup that you’ll be using in production.

This cuts out a huge amount of pain that stems from OS differences. Here are a few examples off the top of my head:

URLField and MacPorts Python 2.5 on OS X. There’s a bug where using verify_exists will crash your site every time you save a model, unless you set a particular environment variable with no debug information. Yeah, I spent a couple of hours tracking that one down at work. Awesome.
Installing PIL on OS X is no picnic. homebrew makes things better, if you use it, so this one isn’t a huge deal.
Every time you update Python in-place on your local machines, ALL of your virtualenvs break because the Python binaries inside are linked against global Python library files. Have fun recreating them. I hope you froze your requirements.txt files before you updated.

Using Vagrant and VMs means you can just worry about ONE operating system and its quirks. It saves you a ton of time.

Aside from that, there’s another benefit to using Vagrant: it strongly encourages you to learn and use an automated provisioning system. Support for Puppet and Chef is built in. I chose Puppet, but if you prefer Chef that’s cool too.

You can also use other tools like Fabric or some simple scripts, but I’d strongly recommend giving Puppet or Chef a fair shot. It’s a lot to learn, but they’re both widely tested and very powerful.

Because you’re developing against a VM and deploying to a VM, you can reuse 90% of the provisioning code across the two.

When I make a new site, I do the following to initialize a new Vagrant VM:

vagrant up (which runs Puppet to initialize the VM)
fab dev bootstrap

When I’m ready to go live, I do the following:

Buy a Linode VPS.
Run Puppet to initialize the VPS.
Enter the Linode info in my fabfile.
fab prod bootstrap

No more screwing around with different paths, different versions of Nginx, different versions of Python. When I’m developing something I can be pretty confident it will “just work” in production without any major surprises.

Using Fabric to Stay Fast and Automate Everything

One of the problems with this setup is that I can’t just run python manage.py whatever any more because I need it to run on the VM.

To get around this I’ve created many simple Fabric tasks to automate the common things I need to do. Fabric is an awesome little Python utility for scripting tasks (like deployments). We use it constantly at Dumbwaiter. Here are a few examples from our fabfiles.

This first set is for running abitrary commands easily.

cmd and vcmd will cd into the site directory on the VM and run a command of my choosing. vcmd will prefix the command with the path to the virtualenv’s bin directory, so I can do something like fab dev vcmd, pip install markdown.

The sdo commands do the same thing, but sudo‘ed.

def cmd(cmd=""):
    '''Run a command in the site directory.  Usable from other commands or the CLI.'''
    require('site_path')

    if not cmd:
        sys.stdout.write(_cyan("Command to run: "))
        cmd = raw_input().strip()

    if cmd:
        with cd(env.site_path):
            run(cmd)

def sdo(cmd=""):
    '''Sudo a command in the site directory.  Usable from other commands or the CLI.'''
    require('site_path')

    if not cmd:
        sys.stdout.write(_cyan("Command to run: sudo "))
        cmd = raw_input().strip()

    if cmd:
        with cd(env.site_path):
            sudo(cmd)

def vcmd(cmd=""):
    '''Run a virtualenv-based command in the site directory.  Usable from other commands or the CLI.'''
    require('site_path')
    require('venv_path')

    if not cmd:
        sys.stdout.write(_cyan("Command to run: %s/bin/" % env.venv_path.rstrip('/')))
        cmd = raw_input().strip()

    if cmd:
        with cd(env.site_path):
            run(env.venv_path.rstrip('/') + '/bin/' + cmd)

def vsdo(cmd=""):
    '''Sudo a virtualenv-based command in the site directory.  Usable from other commands or the CLI.'''
    require('site_path')
    require('venv_path')

    if not cmd:
        sys.stdout.write(_cyan("Command to run: sudo %s/bin/" % env.venv_path.rstrip('/')))
        cmd = raw_input().strip()

    if cmd:
        with cd(env.site_path):
            sudo(env.venv_path.rstrip('/') + '/bin/' + cmd)

This next set is just some common commands that I need to run often.

def syncdb():
    '''Run syncdb.'''
    require('site_path')
    require('venv_path')

    with cd(env.site_path):
        run(_python('manage.py syncdb --noinput'))

def collectstatic():
    '''Collect static media.'''
    require('site_path')
    require('venv_path')

    with cd(env.site_path):
        sudo(_python('manage.py collectstatic --noinput'))

def rebuild_index():
    '''Rebuild the search index.'''
    require('site_path')
    require('venv_path')
    require('process_owner')

    with cd(env.site_path):
        sudo(_python('manage.py rebuild_index'))
        sudo('chown -R %s .xapian' % env.process_owner)

def update_index():
    '''Update the search index.'''
    require('site_path')
    require('venv_path')
    require('process_owner')

    with cd(env.site_path):
        sudo(_python('manage.py update_index'))
        sudo('chown -R %s .xapian' % env.process_owner)

We also use Fabric to automate some of the more complex things we need to do.

This task curl‘s the site’s home page to make sure we haven’t completely borked things. We use it in lots of other tasks as a sanity check.

def check():
    '''Check that the home page of the site returns an HTTP 200.'''
    require('site_url')

    print('Checking site status...')

    if not '200 OK' in local('curl --silent -I "%s"' % env.site_url, capture=True):
        _sad()
    else:
        _happy()

The _happy and _sad functions just print out some simple messages to get our attention:

from fabric.colors import red, green

def _happy():
    print(green('\nLooks good from here!\n'))

def _sad():
    print(red(r'''
          ___           ___
         /  /\         /__/\
        /  /::\        \  \:\
       /  /:/\:\        \__\:\
      /  /:/  \:\   ___ /  /::\
     /__/:/ \__\:\ /__/\  /:/\:\
     \  \:\ /  /:/ \  \:\/:/__\/
      \  \:\  /:/   \  \::/
       \  \:\/:/     \  \:\
        \  \::/       \  \:\
         \__\/         \__\/
          ___           ___           ___           ___
         /__/\         /  /\         /  /\         /  /\     ___
         \  \:\       /  /::\       /  /:/_       /  /:/_   /__/\
          \  \:\     /  /:/\:\     /  /:/ /\     /  /:/ /\  \  \:\
      _____\__\:\   /  /:/  \:\   /  /:/ /:/_   /  /:/ /::\  \  \:\
     /__/::::::::\ /__/:/ \__\:\ /__/:/ /:/ /\ /__/:/ /:/\:\  \  \:\
     \  \:\~~\~~\/ \  \:\ /  /:/ \  \:\/:/ /:/ \  \:\/:/~/:/   \  \:\
      \  \:\  ~~~   \  \:\  /:/   \  \::/ /:/   \  \::/ /:/     \__\/
       \  \:\        \  \:\/:/     \  \:\/:/     \__\/ /:/          __
        \  \:\        \  \::/       \  \::/        /__/:/          /__/\
         \__\/         \__\/         \__\/         \__\/           \__\/

         Something seems to have gone wrong!
         You should probably take a look at that.
    '''))

This one is for when python manage.py reset APP is broken because you’ve changed some db_column names and Django chokes because of some constraits and you just want to reset the fucking app.

It’s the “NUKE IT FROM ORBIT!!” option.

def KILL_IT_WITH_FIRE(app):
    require('site_path')
    require('venv_path')

    with cd(env.site_path):
        # Generate and download the reset SQL.
        sudo(_python('manage.py sqlreset %s > reset.orig.sql' % app))
        get('reset.orig.sql')

        with open('reset.sql', 'w') as f:
            with open('reset.orig.sql') as orig:
                # Step through the first chunk of the file (the "drop" part).
                line = orig.readline()
                while not line.startswith('CREATE'):
                    if 'CONSTRAINT' in line:
                        # Don't write out CONSTRAINT lines.
                        # They're a problem when you change db_colum names.
                        pass
                    elif 'DROP TABLE' in line:
                        # Cascade drops.
                        # Hence with "with fire" part of this task's name.
                        line = line[:-2] + ' CASCADE;\n'
                        f.write(line)
                    else:
                        # Write other lines through untoched.
                        f.write(line)
                    line = orig.readline()

                # Write out the rest of the file untouched.
                f.write(line)
                f.write(orig.read())

    # Upload the processed SQL file.
    put('reset.sql', os.path.join(env.site_path, 'reset.ready.sql'), use_sudo=True)

    with cd(env.site_path):
        # Use the SQL to reset the app, and fake a migration.
        run(_python('manage.py dbshell < reset.ready.sql'))
        sudo(_python('manage.py migrate --fake --delete-ghost-migrations ' + app))

This task uses Mercurial’s local tags to add a production or staging tag in your local repository, so you can easy see where the production/staging servers are at compared to your local repo.

def retag():
    '''Check which revision the site is at and update the local tag.

    Useful if someone else has deployed (which makes your production/staging local
    tag incorrect.
    '''
    require('site_path', provided_by=['prod', 'stag'])
    require('env_name', provided_by=['prod', 'stag'])

    with cd(env.site_path):
        current = run('hg id --rev . --quiet').strip(' \n+')

    local('hg tag --local --force %s --rev %s' % (env.env_name, current))

This task tails the Gunicorn logs on the server so you can quickly find out what’s happening when things blow up.

def tailgun(follow=''):
    """Tail the Gunicorn log file."""
    require('site_path')

    with cd(env.site_path):
        if follow:
            run('tail -f .gunicorn.log')
        else:
            run('tail .gunicorn.log')

We’ve got a lot of other tasks but they’re pretty specific to our setup.

Wrangling Databases with South

If you’re not using South, you need to start. Now.

No, really, I’ll wait. Take 30 minutes, try the tutorial, wrap your head around it and come back. It’s far more important than this blog post.

Useful Fabric Tasks

South is awesome but its commands are very long-winded. Here’s the set of fabric tasks I use to save quite a bit of typing:

def migrate(args=''):
    '''Run any needed migrations.'''
    require('site_path')
    require('venv_path')

    with cd(env.site_path):
        sudo(_python('manage.py migrate ' + args))

def migrate_fake(args=''):
    '''Run any needed migrations with --fake.'''
    require('site_path')
    require('venv_path')

    with cd(env.site_path):
        sudo(_python('manage.py migrate --fake ' + args))

def migrate_reset(args=''):
    '''Run any needed migrations with --fake.  No, seriously.'''
    require('site_path')
    require('venv_path')

    with cd(env.site_path):
        sudo(_python('manage.py migrate --fake --delete-ghost-migrations ' + args))

Remember that running a migration without specifying an app will migrate everything, so a simple fab dev migrate will do the trick.

Watching for Changes

When developing locally you’ll want to make a change to your code and have the server reload that code automatically. The Django development server does this, and we can hack it into our Vagrant/Gunicorn setup too.

First, add a monitor.py file at the root of your project (I believe I found this code here, but I may be wrong):

import os
import sys
import time
import signal
import threading
import atexit
import Queue

_interval = 1.0
_times = {}
_files = []

_running = False
_queue = Queue.Queue()
_lock = threading.Lock()

def _restart(path):
    _queue.put(True)
    prefix = 'monitor (pid=%d):' % os.getpid()
    print >> sys.stderr, '%s Change detected to \'%s\'.' % (prefix, path)
    print >> sys.stderr, '%s Triggering process restart.' % prefix
    os.kill(os.getpid(), signal.SIGINT)

def _modified(path):
    try:
        # If path doesn't denote a file and were previously
        # tracking it, then it has been removed or the file type
        # has changed so force a restart. If not previously
        # tracking the file then we can ignore it as probably
        # pseudo reference such as when file extracted from a
        # collection of modules contained in a zip file.

        if not os.path.isfile(path):
            return path in _times

        # Check for when file last modified.

        mtime = os.stat(path).st_mtime
        if path not in _times:
            _times[path] = mtime

        # Force restart when modification time has changed, even
        # if time now older, as that could indicate older file
        # has been restored.

        if mtime != _times[path]:
            return True
    except:
        # If any exception occured, likely that file has been
        # been removed just before stat(), so force a restart.

        return True

    return False

def _monitor():
    while 1:
        # Check modification times on all files in sys.modules.

        for module in sys.modules.values():
            if not hasattr(module, '__file__'):
                continue
            path = getattr(module, '__file__')
            if not path:
                continue
            if os.path.splitext(path)[1] in ['.pyc', '.pyo', '.pyd']:
                path = path[:-1]
            if _modified(path):
                return _restart(path)

        # Check modification times on files which have
        # specifically been registered for monitoring.

        for path in _files:
            if _modified(path):
                return _restart(path)

        # Go to sleep for specified interval.

        try:
            return _queue.get(timeout=_interval)
        except:
            pass

_thread = threading.Thread(target=_monitor)
_thread.setDaemon(True)

def _exiting():
    try:
        _queue.put(True)
    except:
        pass
    _thread.join()

atexit.register(_exiting)

def track(path):
    if not path in _files:
        _files.append(path)

def start(interval=1.0):
    global _interval
    if interval < _interval:
        _interval = interval

    global _running
    _lock.acquire()
    if not _running:
        prefix = 'monitor (pid=%d):' % os.getpid()
        print >> sys.stderr, '%s Starting change monitor.' % prefix
        _running = True
        _thread.start()
    _lock.release()

Next add a post_fork hook to your Gunicorn config file that uses the monitor to watch for changes:

def post_fork(server, worker):
    import monitor
    import local_settings
    if local_settings.DEBUG:
        server.log.info("Starting change monitor.")
        monitor.start(interval=1.0)

Now the Gunicorn server will automatically restart whenever code is changed. Use whatever method for determining debug status that you like. We use local_settings.py files which all have DEBUG variables, so that works for us.

It will not restart when you add new code (e.g. when you install a new app), so you’ll need to handle that manually with fab dev restart, but that’s not too bad!

Using the Werkzeug Debugger with Gunicorn

The final piece of the puzzle is being able to use the fantastic Werkzeug Debugger while running on the development VM with Gunicorn.

To do this, create a debug_wsgi.py file at the root of your project:

import os
import sys
import site

parent = os.path.dirname
site_dir = parent(os.path.abspath(__file__))
project_dir = parent(parent(os.path.abspath(__file__)))

sys.path.insert(0, project_dir)
sys.path.insert(0, site_dir)

site.addsitedir('VIRTUALENV_SITE_PACKAGES')

from django.core.management import setup_environ
import settings
setup_environ(settings)

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

from werkzeug.debug import DebuggedApplication
application = DebuggedApplication(application, evalex=True)

def null_technical_500_response(request, exc_type, exc_value, tb):
    raise exc_type, exc_value, tb
from django.views import debug
debug.technical_500_response = null_technical_500_response

Have Gunicorn use this file to run your development server with gunicorn debug_wsgi:application.

Make sure to replace 'VIRTUALENV_SITE_PACKAGES' with the full path to your virtualenv’s site_packages directory. You might want to make this a setting in a machine-specific settings file.

Pulling Uploads

Once you give a client access to a site they’ll probably be uploading images (through Django’s built-in file uploading features or with django-filebrowser).

When you’re making changes locally it’s often useful to have these uploaded files on your local VM, otherwise you end up with a bunch of broken images.

Here’s a simple Fabric task that will pull down all the uploads from the server:

def pull_uploads():
    '''Copy the uploads from the site to your local machine.'''
    require('uploads_path')

    sudo('chmod -R a+r "%s"' % env.uploads_path)

    rsync_command = r"""rsync -av -e 'ssh -p %s' %s@%s:%s %s""" % (
        env.port,
        env.user, env.host,
        env.uploads_path.rstrip('/') + '/',
        'media/uploads'
    )
    print local(rsync_command, capture=False)

You might be wondering about the line that strips / characters and then adds them back in. rsync does different things depending on whether you end a path with a /, so this is actually pretty important.

In your host task you’ll need to set the uploads_path variable to something like this:

import os
env.site_path = os.path.join('var', 'www', 'myproject')
env.uploads_path = os.path.join(env.site_path, 'media', 'uploads')

Now you can run fab production pull_uploads to pull down all the files people have uploaded to the production server.

Preventing Accidents

Deploying to test and staging servers should be quick and easy. Deploying to production servers should be harder to prevent people from accidentally doing it.

I’ve created a little function that I call before deploying to production servers. It forces me to type in random words from the system word list before proceeding to make sure I really know what I’m doing:

import os, random

from fabric.api import *
from fabric.operations import prompt
from fabric.utils import abort

WORDLIST_PATHS = [os.path.join('/', 'usr', 'share', 'dict', 'words')]
DEFAULT_MESSAGE = "Are you sure you want to do this?"
WORD_PROMPT = '  [%d/%d] Type "%s" to continue (^C quits): '

def prevent_horrible_accidents(msg=DEFAULT_MESSAGE, horror_rating=1):
    """Prompt the user to enter random words to prevent doing something stupid."""

    valid_wordlist_paths = [wp for wp in WORDLIST_PATHS if os.path.exists(wp)]

    if not valid_wordlist_paths:
        abort('No wordlists found!')

    with open(valid_wordlist_paths[0]) as wordlist_file:
        words = wordlist_file.readlines()

    print msg

    for i in range(int(horror_rating)):
        word = words[random.randint(0, len(words))].strip()
        p_msg = WORD_PROMPT % (i+1, horror_rating, word)
        answer = prompt(p_msg, validate=r'^%s$' % word)

You may need to adjust WORDLIST_PATHS if you’re not on OS X.

Working with Third-Party Apps

One of the best parts about working with Django is that many problems have already been solved and the solutions have been released as open-source applications.

We use quite a few open-source apps, and there are a couple of tricks I’ve learned to make working with them easier.

Installing Apps from Repositories

If I’m going to use an open-source Django app in a project I’ll almost always install it as an editable repository on the VM with pip install -e.

Others may disagree with me on this, but I think it’s the best way to work.

Often I’ll find a bug that I think may be in one of the third-party apps I’m using. Installing the apps as repositories makes it easy to read their source and figure out if the bug is really in the app.

If the bug is in the third-party app having the app installed as a repository makes it simple to fix the bug, fork the project on BitBucket or GitHub, send a pull request, and get back to work.

Mirroring Repositories

One problem we’ve run into at Dumbwaiter is that the repos for third-party apps we use are scattered across GitHub, BitBucket, Google Code, and other servers. If any one of these services goes down we’re stuck waiting for it to come back up.

A while ago I took half a day and consolidated all of these repos onto one of the servers that we control. The basic process went like this:

Use hg-git and hgsubversion to convert the git and SVN repos to Mercurial repos.
Set up a master mirror Mercurial repo with all the app repos as subrepos.
Push the master repo and all the subrepos up to one of our Linodes.

Now we can use -e ssh://hg@OUR_LINODE/mirror/APP@REV_THAT_WORKS#egg=APP in our requirements.txt files to install apps from our mirror. When we want to update our dependencies we can simply pull from the upstream repos and commit in the mirror repo.

If our mirror goes down it’s not a big deal, because we have far bigger problems to worry about than new projects.

I wrote a few scripts to automate updating apps and such, but they’re extremely hacky so I don’t want to post them here. Take half a day and write your own set — it’s definitely worth it to have your own mirror of your specific dependencies.

Using BCVI to Edit Files

I said that when I find a bug that I think is in a third-party app I’ll poke around with the app and try to figure it out. But since all the apps are installed in a virtualenv on the Vagrant VM it might seem like it’s a pain in the ass to edit those files!

Luckily BCVI exists. It’s a utility that opens a “back channel” to your local machine when you SSH and lets you run vi FILE to open that file in Vim/MacVim/GVim/etc on your local machine. When you save the file it uploads it back to the server automatically for you.

It can be a bit tricky to set up, but it’s worth it. Trust me.

Improving the Admin Interface

I’m going to be honest: Django’s admin interface is the main reason I’m still using it. Other frameworks like Flask are great, but Django’s admin saves me ridiculous amounts of time when I’m making simple CRUD sites for clients.

That said, the Django admin isn’t the prettiest thing around, but we can give it a facelift.

Enter Grappelli

Grappelli is a Django app that reskins the admin interface beautifully. It also adds some functionality like drag-and-drop reordering of inlines, and allows you to customize the dashboard to your liking. Every Django site I work on uses Grappelli — it’s just that good.

The downside of Grappelli is that it changes quite a lot and breaks backwards compatibility at the drop of a hat.

If you’re going to use Grappelli you must freeze your requirements.txt files and work with a single version at a time. Trying to always work from the trunk will make you drink.

An Ugly Hack to Show Usable Foreign Key Fields

A limitation of both Grappelli and the stock Django admin is that it seems like you can’t easily show fields from related models in the admin list view.

For example, if you’re new to Django you might expect this to work:

class BlogEntryAdmin(admin.ModelAdmin):
    list_display = ('title', 'author__name')

Unfortunately Django chokes on the author__name lookup. You can display the name without too much fuss:

class BlogEntryAdmin(admin.ModelAdmin):
    list_display = ('title', 'author_name')

    def author_name(self, obj):
        return obj.name

That will display the name just fine. However, it won’t be a fully-fledged column in the Django admin because you can’t sort on it.

It may seem like this is the end — if it could be a fully-functional field, why wouldn’t Django just let you use author__name? Luckily we can add one more line to fix the problem:

class BlogEntryAdmin(admin.ModelAdmin):
    list_display = ('title', 'author_name')

    def author_name(self, obj):
        return obj.name
    author_name.admin_order_field = 'author__name'

Now the author name has all the functionality of a real list_display entry.

Using Django-Annoying

If you haven’t heard of django-annoying you should definitely check it out. It’s got a bunch of miscellaneous functions that fix some common, annoying parts of Django.

My two personal favorites from the package are a pair of decorators that help make your views much, much cleaner.

The render_to Decorator

The decorator is called render_to and it eliminates the ugly render_to_response calls that Django normally forces you to use in every single view.

Normally you’d use something like this:

def videos(request):
    videos = Video.objects.all()
    return render_to_response('video_list.html', { 'videos': videos },
                              context_instance=RequestContext(request))

With render_to your view gets much cleaner:

@render_to('video_list.html')
def videos(request):
    videos = Video.objects.all()
    return { 'videos': videos }

Less typing context_instance=... over and over, and less syntax to remember.

Yes, I know about Django 1.3’s render shortcut. You have to type request every single time with render, so the render_to decorator still wins.

The ajax_request Decorator

The ajax_request decorator is like render_to for AJAX requests. You simply return a Python dictionary from your view and the decorator handles the JSON encoding and such:

@ajax_request
def ajax_get_entries(request):
    blog_entries = BlogEntry.objects.all()
    return { 'entries': [(entry.title, entry.get_absolute_url())
                         for entry in entries]}

Templating Tricks

I’m not a frontend developer, but I’ve done my share of HTML hacking at Dumbwaiter. Here are a few of the tricks I’ve learned.

Null Checks and Fallbacks

A common pattern I see in Django templates looks like this:

{% if business.title %}
    {{ business.title }}
{% else %}
    {{ business.short_title }}
{% endif %}

Here’s a simpler way to do that:

{% firstof business.title business.short_title %}

firstof will return the first non-Falsy item in its arguments.

Manipulating Query Strings

Query strings are normally not a big deal, but every once in a while you’ll have a model listing page where you need to filter by category, and number of spaces, and tags, etc all at once.

If you’re trying to manage GET queries manually it can get pretty hairy very fast.

This Django snippet makes working with query strings in templates a breeze.

Satisfying Your Designer with Typogrify

If you haven’t heard of Typogrify you should take a look at it. It makes it easy to add all the typographic goodness your designers are looking for.

The Flat Page Trainwreck

Creating a site for a client is very different than creating a site for yourself. For pretty much every client we’ve dealt with we’ve heard: “can’t we just create a new page at /drink-special/ for this special deal we’re running?”

Having clients go through you to make new pages is simply too much overhead. We needed a way to let clients create new pages (like /drink-special/) on the fly, without our intervention.

Django has a “flatpages” app that solves this problem. Kind of.

When using flat pages clients need to do two things that are often too much for non-technical people:

Manage URLs manually.
Write all content as raw HTML in a single text field.

We’ve tried a lot of Django CMS apps at Dumbwaiter, and none of them made us happy. They all seemed to have some or all of the following problems:

They take over your site and make you write a “Django-WhateverCMS site” instead of a “Django site”.
They’re extremely feature-rich and complicated with features like internationalization, redirects, versions, and many others. This is great if you need the flexibility, but bad if your clients just need to create a couple of pages.
They break APPEND_TRAILING_SLASH and make you clutter your urls.py files with a bunch of extra code ot handle this.

I finally got fed up and wrote my own Django CMS app: Stoat. Stoat is designed to be sleek, with only the features that our clients need.

It’s not officially version 1.0 yet, but we’re using it for a few clients and it’s working well. Check it out if you’re looking for a more lightweight CMS app.

Editing with Vim

I use Vim to edit everything. Naturally I’ve found a bunch of plugins, mappings and other tricks that make it even better when working on Django projects.

Vim for Django

There are a lot of ways to make Vim work with Django. I won’t go into all of them in this post, but a good place to start is this Django wiki page.

Filetype Mappings

Most files in a Django project have one of two extensions: .py and .html. Unfortunately these extensions aren’t unique to Django, so Vim doesn’t automatically set the correct filetype when you open one.

I’ve added a few mappings to my .vimrc to make it quick and easy to set the correct filetype:

nnoremap _dt :set ft=htmldjango<CR>
nnoremap _pd :set ft=python.django<CR>

I also have a few autocommands that set the filetype for me when I’m editing a file whose name “sounds like” a Django file:

au BufNewFile,BufRead admin.py     setlocal filetype=python.django
au BufNewFile,BufRead urls.py      setlocal filetype=python.django
au BufNewFile,BufRead models.py    setlocal filetype=python.django
au BufNewFile,BufRead views.py     setlocal filetype=python.django
au BufNewFile,BufRead settings.py  setlocal filetype=python.django
au BufNewFile,BufRead forms.py     setlocal filetype=python.django

Python Sanity Checking

Lets be honest here: it takes a lot of work to turn Vim into an “IDE”, and even then it doesn’t reach the level of something like Eclipse for Java. Anyone who claims it has the same levels of integration and functionality is simply lying.

With that said I’ll make an opinionated statement that is going to piss some of you off.

I am a programmer, not an IDE operator.

I know Python.

I know Django.

I don’t need to hit Cmd+Space twice for every line of code I write.

When someone asks me “how do you run your site” I do not answer: “click the green triangle in Eclipse”.

However, I am human. I do stupid things like forgetting a colon or forgetting an import. To help me with those problems I’ve turned to Syntastic and Kevin Watters’ Pyflakes fork for Vim.

Syntastic is a Vim plugin that adds on-the-fly syntax-checking for many different file formats. If you have Pyflakes installed it will automatically show you errors in your code.

Pyflakes doesn’t have IDE-level integration with your code. It doesn’t check that whatever libraries you import actually exist. It simply checks that your files are probably-valid Python, and tells you when they’re not.

This is enough for me. It catches the stupid mistakes I make. The less-stupid, more-subtle mistakes slip by it, but to be fair many of them would have slipped by an “IDE” as well.

Javascript Sanity Checking and Folding

Syntastic also supports Javascript if you have Javascript Lint installed (brew install jsl on OS X). It’s not perfect but it will catch things like using trailing commas in object literals.

Some people like using CTags to get an overview of their code. I take a more low-tech approach and am in love with code folding. When I fold my code I automatically get an overview of everything in each file.

By default Vim doesn’t fold Javascript files, but you can add some basic, perfectly serviceable folding with these two lines in your .vimrc:

au FileType javascript setlocal foldmethod=marker
au FileType javascript setlocal foldmarker={,}

Django Autocommands

I rarely work with raw HTML files any more. Whenever I open a file ending in .html it’s almost always a Django template (or a Jinja template, which has a very similar syntax). I’ve added an autocommand to automatically set the correct filetype whenever I open a .html file:

au BufNewFile,BufRead *.html setlocal filetype=htmldjango

I also have some autocommands that tweak how a few specific files are handled:

au BufNewFile,BufRead urls.py      setlocal nowrap
au BufNewFile,BufRead settings.py  normal! zR
au BufNewFile,BufRead dashboard.py normal! zR

This automatically unfolds urls.py, dashboard.py and settings.py (I prefer seeing those unfolded) and unsets line wrapping for urls.py (lines in a urls.py file can get long and are hard to read when wrapped).

Conclusion

I hope that this longer-than-expected blog entry has given you at least one or two things to think about.

I’ve learned a lot while working with Django for Dumbwaiter every day, but I’m sure there’s still a lot I’ve missed. If you see something I could be doing better please let me know!

Going Paper-Free for $220

2011-05-26T13:44:00Z

It’s 2011. Personal computers have been around and popular for well over a decade now, and yet we still have to deal with a huge amount of physical paper.

I’ve been wanting to go paper-free for a long time now. The advantages are obvious:

Paper takes up physical space in our homes that digital files don’t.
Digital files, if properly encrypted, are far more secure than sheets of paper that could be stolen.
Digital files can be searched in an instant, while papers have to be laboriously sorted through.
Digital files can be backed up perfectly and easily.

After reading this article I was psyched to scan and shred all the boxes of paper sitting in my apartment, but the $420+ price tag was hard to swallow. I started looking around for other options.

Here are the requirements I have for any paper-free system:

The scanned files need to be OCR’ed so I can search them easily. I’m too lazy to categorize and tag files manually.
I need to be able to scan files anywhere. If I’m out at dinner I want to be able to snap a picture of my receipt and tear it up right there.
No “cloud” services allowed for unencrypted important documents. I simply don’t trust Google/Dropbox/etc enough to put my bank statements and such there.
Files need to be backed up securely in case my apartment burns down.
The entire process needs to be automated as much as possible, otherwise I’ll get lazy and not scan things.

It’s taken me a while, but I’ve finally got a system I’m happy with. This post will describe each part and how they fit together. The total cost is about $220, $160 of which is for a physical scanner.

Note: I use OS X and an iPhone, so this post will focus on that platform. However, the important pieces of software will run on Windows and I’m sure there are Windows/Android equivalents to the other pieces.

Scanning at Home
Scanning on the Go
OCR’ing Scanned Documents
Gluing Everything Together
Backing Up
Destroying the Originals
Summary

Scanning at Home

The first step to becoming paper-free is obviously scanning your documents. There are a lot of scanners out there, some more expensive than others. I eventually settled on a Doxie for $160.

I chose the Doxie because:

It’s compact.
It runs with a single USB cable.
It’s cross-platform.
Its software has a great, polished UI.
It has a “multiple-function button” that lets you control it without the mouse/keyboard.

The first, second, and last points mean that (with a USB extension cable) I can scan documents while sitting on the couch and watching Netflix, which is critical for lazy people like me.

I set Doxie to save scans on my Desktop. The scanning process is pretty simple so I won’t describe it here. Check out Doxie’s documentation for more information.

Note: When I first received my Doxie and tried to calibrate it, it simply made a grinding noise and wouldn’t feed the paper. I emailed their tech support and within half an hour I got a response back saying they were shipping me a replacement immediately.

When I got the replacement it worked like a charm. Their customer service was so great that I’d still recommend the Doxie even though my first one was a dud.

Scanning on the Go

As I mentioned before, I want to be able to scan things while out and about with my iPhone. There are a bunch of iPhone document-scanning apps out there. I settled on JotNot for $7 because it has a decent UI and supports multiple-page PDFs.

JotNot’s UI is pretty easy to get the hang of so I won’t go over it here.

Once I finish scanning something I send the PDF to a Dropbox folder called “JotNot”.

I know I said in my requirements that “cloud” services weren’t allowed, but I make an exception for non-critical things that I’d be scanning with my phone. I don’t care if Dropbox knows how much I spent on dinner.

OCR’ing Scanned Documents

The next step is to run the scanned PDFs through an OCR program so they can be searched with Spotlight.

I looked at a lot of OCR software and finally settled on PDF OCR X for $30. It has a simple interface, does a pretty good job at OCR’ing, has a free version so I could try it out, and is cross-platform.

Using it is simple: you drag a PDF onto the app and select your desired settings (make sure to choose “searchable PDF” as the output format). The app will think for a while and then create a new PDF next to the old one with the searchable text embedded.

Once you’ve done this once you should go into the preferences and change it to non-interactive mode so that it won’t prompt you for the settings every time you use it.

Gluing Everything Together

So far we’ve got two folders with scanned PDFs and a method for OCR’ing them. The next step is to automate the process.

I use an app called Hazel to do this. It’s $21 for a license and well worth it. We’ll set up four rules to make our lives easier.

Before we start we need to create two folders somewhere (you can name them whatever you like):

Pending OCR: A folder to hold documents that are waiting to be OCR’ed.
Dead Trees: A folder to hold the final, OCR’ed versions of our documents.

The first rule watches the Desktop for scans from Doxie. Any files placed on the Desktop whose name starts with “Doxie Doc” will be renamed to include the current date and time, and then moved to the “Pending OCR” folder.

Note:: you’ll need to click the date created bubble and then “Edit Date” to get the time as well as the date into the filename.

The second rule watches the “JotNot” folder for scans from the iPhone app. Any PDFs that appear in here (i.e. that are synced down from Dropbox) will be moved to the “Pending OCR” folder. We don’t need to rename them like we did with the Doxie scans because JotNot already includes the date and time of scans in the filenames by default.

Now that we’ve got all of our scans going into the same folder (with unique names) we can set up a rule to OCR them. The third rule watches the “Pending OCR” folder for PDFs. When a PDF lands in the folder it will be moved to its final destination folder (“Dead Trees” in my case) and then opened in PDF OCR X. Because I’ve put PDF OCR X in non-interactive mode the files will automatically be OCR’ed without any intervention from me.

The fourth and final rule watches for the OCR’ed copies of our scans and runs a script to move the originals to the trash once the searchable versions are ready. It doesn’t delete the files completely because I want a safety net in case something goes wrong.

Note: make sure you change the Shell to /usr/bin/python. Here’s the text of the script so you can copy and paste it:

import sys, os

RM_CMD = r"""osascript -e 'tell app "Finder" to move the POSIX file "%s" to trash'"""
old_file = sys.argv[1].rsplit('.', 2)[0]
if os.path.exists(old_file):
    os.system(RM_CMD % os.path.abspath(old_file))

Once these four rules are in place we can simply scan a document with Doxie or JotNot and it will automatically be OCR’ed and placed in our “Dead Trees” folder, with no intervention from us!

Backing Up

A while ago I was using Mozy for full backups. Recently they changed their pricing so it was no longer unlimited. When that happened I switched to Backblaze and couldn’t be happier.

Backblaze’s UI is leaps and bounds above Mozy’s, and they offer an option to generate a secure encryption key for encrypting your backups. I highly recommend this, but be sure to have a few copies of your key because you’ll need it to restore your backups.

Backblaze is also only $5 per month (less if you pay for a year in advance) for unlimited backups which is definitely a bargain. As a bonus, they just released a “find my computer” feature that’s kind of like a lightweight version of Undercover, so it’s an even better deal.

Destroying the Originals

Once the documents are scanned and backed up it’s time to destroy the physical paper. If you live in a rural area you could burn them for free.

Those of us that can’t start random fires need a paper shredder. I use a shredder I picked up a long time ago — any crosscut shredder will do the job.

Summary

After all of this I’ve now got a mostly-automated system that lets me go paper-free. The costs are:

Doxie Scanner: $160
JotNot: $7
PDF OCR X: $30
Hazel: $21
Backblaze: $5 per month

For me the $218 initial cost is worth it. Now I can search all of my paper in a instant and my apartment is much less cluttered. If you have the money to spare I’d definitely consider trying it.

On Learning and Teaching

2011-05-22T17:00:00Z

I recently read a blog entry by Zack Kurmas called “The deep end of the pool“. In it he talks about why he thinks some students succeed in introductory Computer Science classes while others fail. If you’re a programmer and haven’t read it, stop right now and read it.

In a nutshell: I think he’s pretty much correct. I wanted to write this post because I think I have a solution to the problem that might work, at least for some people.

Background
Similarities in Dancing and Programming
Plateaus in Learning
Getting Through Plateaus
Conclusion

Background

I’m a computer programmer by trade. I went to RIT for five years to get my degree in Computer Science and before that I was working with toy programs during high school.

While I was at RIT I started going to Swing Dance Club. I learned to swing dance there about eight years ago. A few years later I started I began teaching Swing Dance Club with some of my friends, and about a year ago I started teaching Blues Dancing with Lady Luck Blues.

I have a couple of other semi-serious hobbies that I’ve taken classes in as well: photography and bass playing.

To summarize: I’ve had experience in a bunch of different areas, and I’ve been teaching one of them for a while.

Disclaimer: one thing I’ve never studied is teaching itself, so feel free to take everything I say about it with a large grain of salt. I haven’t done any hard research and I could be completely wrong.

Similarities in Dancing and Programming

In his article, Zack talks about programming syntax being a stumbling block for beginners. I think he’s very right and I think there’s a similar concept in dancing.

Programming Syntax

If you’re a programmer you can skip this section.

For the non-programmers: “syntax” is the combination of letters, numbers and symbols that you write to create computer programs. For example, look at the following two bits of text:

if x == 2:
    print "Hello"
x = 10

if x == 2:
    print 'Hello"
x = 10

Notice the difference? The first snippet says "Hello" while the second says 'Hello". Notice how the first uses two double quotes (") while the second uses one double and one single quote (').

The first bit of code will work, but the second one won’t. Little differences like these might not seem like a big deal, but to a computer they’re as different as abcdefg and o34%^8@@.com.

Dancing Footwork

If you’re a dancer you can skip this section.

For the non-dancers: “footwork” is the pattern of steps you make when you dance. For example, in Lindy Hop (a style of swing dancing) the basic footwork looks like this:

Beats:   1   2   3   4   5   6   7   8
Steps:   x   x   x  xx   x   x   x  xx

For East Coast Swing, the basic footwork looks like this:

Beats:   1   2   3   4   5   6
Steps:   x   x   x  xx   x  xx

Notice the differences? Lindy Hop footwork is based on an eight beat pattern, while East Coast footwork is based on six beats. The patterns also have the “extra steps” in different places.

When you’re dancing with a partner you both need to be doing the same footwork, otherwise you can’t really dance together.

Footwork is Syntax and Syntax is Footwork

When you’re learning both programming and dancing there are lots of deep, high-level concepts you can tackle. For programming you have things like object orientation, recursion, and API design. In dancing you have things like musicality, listening to your follower, and negative space.

However: in my experience teaching dancing you need to learn and understand the footwork before you can learn the higher concepts. If you’re still thinking about where to put your feet when you’re dancing you won’t have the spare brainpower to think about those other concepts.

In Zack’s post he says the same thing about programming. If you’re still struggling with understanding the syntax of programs you can’t simultaneously pay attention to the higher concepts that are being taught. I’ve never taught programming but my personal observations of programmers make me think he’s right.

To be clear: there aren’t just two levels (low and high). There are many levels/layers, each of which builds on the last. Also: the layers are not linear — learning concept A might allow you to learn either B or C.

However: the same idea applies most of the time. You need to learn and internalize lower levels before moving on to the higher ones.

Plateaus in Learning

One thing I’ve noticed while learning dancing, programming, photography, and other skills is that my learning seems to progress quickly at times and slowly at others.

When I’m learning quickly everything is great. I feel good about myself and my abilities and I throw myself into learning as much as I can.

On the other hand when I’m learning slowly (hitting a “plateau”) I often get frustrated or discouraged. Nothing I try seems to speed up my ascent out of these plateaus. It seems like the only thing that works is time and practice of what I already know.

I think the reason for this is that once I learn a bunch of new concepts my brain needs some time to process them. I can learn the concepts and use them if I concentrate, but it’s not effortless. I think the reason I finally start learning quickly again after some time is that I’ve internalized the previous concepts and now I can move on to more difficult ones which build on the previous ones.

The first plateau of programming is the syntax and the first plateau of dancing is footwork. The bad part about this is that dancing when you only know footwork or programming when you only know syntax isn’t much fun. You can’t do all of the most interesting things that make these skills so rewarding.

Getting Through Plateaus

In my own experience the hardest part about teaching is getting students through the plateaus. Because it just takes time and practice to process what they’ve already learned there’s not much I can do to help them. Students take more advanced classes but don’t really grab them until they’re fluent in the things they’ve previously learned.

As a teacher, this is incredibly frustrating. I want to teach my students what I know, but I know they won’t absorb it until they’ve absorbed the previous lessons.

The Lie and the Solution

Here’s the kicker, and what I think might provide the solution to this problem: I lied when I said that the only things that help me through plateaus are time and practice.

There’s one more thing that helps me get through plateaus: connecting the concepts I’m trying to internalize with concepts from other skills that I already know.

An example of this that I’ve written about before is negative space. When I was learning blues dancing I realized that “stopping your movement” was simply a different expression of the concept of “negative space” in photography.

I had already struggled through internalizing the concept of negative space in photography, so when I connected this with dancing it helped make things “click” together and made the plateau shorter.

The Problem with the Solution

In my own life this solution is great.

I have a bunch of hobbies and all of them mesh together in different ways. Sometimes I can connect what I know in one to what I know in another and get through these plateaus more quickly. When I can’t do that I can focus on one of my other hobbies while practicing another, which reduces the frustration of not progressing faster.

When teaching students, however, this isn’t a trick I can use effectively.

I can’t really tell a student: “you need to find something else to focus on for a while while you internalize what I’ve taught you.” If they want to learn dancing, they’ll feel like taking more dance classes is the way to go. They’re paying me to teach them so I obviously have to try to teach them.

I sometimes try to teach the “analogies” between skills that have helped me. Sometimes this works well — the “Negative Space in Dancing” class that my partner and I have taught seems to go over well with most of the students.

The problem is that everyone has different skills. Trying to connect a concept in programming to a concept in photography is futile if the student doesn’t understand the photographic concept.

The Solution to the Problem with the Solution

Disclaimer: once again, I’ve never studied teaching itself, so this might not be a new idea and/or I could be completely wrong about this.

I’ve thought a long time about a way to fix this. For students that are simply taking a class here and there with me, I haven’t found a good solution.

However, for students enrolled in a longer series of classes, like college students studying a major, I think there might be a way to use this idea (linking concepts from disparate skills) to help them learn.

When I went to RIT I didn’t just take Computer Science classes. I had to take a few science classes, a few math classes, a few history classes, and so on. I believe other colleges work on the same system: you take many classes in your major and one or two classes in everything else. The idea is that you become a “well-rounded” person.

The bad part about this process is that one or two classes is not enough to get to one of these “plateaus”, struggle through it, and continue on. Therefore you don’t have time to internalize the skills you learn, which means that you can’t connect them to concepts in your major.

I’m also not sure how effective these random classes are at making students “well-rounded”. If you don’t get far enough to internalize the skills you learn in a class are they really useful to you once you’re done with those classes?

I think the way around this is to ditch the “one major and a tiny bit of everything else” process and move to a “one major and two or three minors” process.

This will help students past the plateaus that discourage them, because they’ll have learned enough in their “minors” to be able to connect concepts to those in their major. Teachers might not be able to teach specific analogies because the students will all have different minors, but each student will be able to connect the concepts in their majors on their own to whatever they happen to know from their minors.

This process also has another benefit: students will leave college retaining more skills that they would have otherwise. A programming student could leave college with a real, effective amount of knowledge in music; a writing major could leave with a real, effective amount of knowledge in biology, and so on.

Conclusion

I’m not naive enough to think that any college is going to change their entire curriculum based on some random guy’s blog post.

My intended audience for this post is not colleges, but people that want to learn skills faster and more effectively. I know that there are a lot of people out there hungry for knowledge and I hope that some of them can benefit from the ideas I’ve given, learn faster, and make the world a better place.