The Daily Build

Getting Scalatest and Scalacheck to Work

2014-01-29T22:50:00-05:00

ScalaTest is a decent library for testing scala code. It integrates with ScalaCheck for "property based testing". In this type of testing, you specify properties of a class or method, and the library generates test cases. I haven't done much property based testing, and it sounds interesting, so I thought I'd give it a shot.

Unfortunately, this two-year-old commit deprecated a ScalaCheck API that is still in use by the current release of ScalaTest. This means that following the instructions on the respective websites for integrating the code into your build.sbt are wrong. So the following will not work:

libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.11.3" % "test"

libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.0" % "test"

When you try to use certain parts of ScalaTest, you will get this message at compile time:

> test
[info] Compiling 1 Scala source to /home/brian/projects/check/target/scala-2.10/test-classes...
[error] bad symbolic reference. A signature in Configuration.class refers to type Params
[error] in object org.scalacheck.Test which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling Configuration.class.
[error] one error found
[error] (test:compile) Compilation failed

There's an open issue on ScalaTest's github, and there's a snapshot release that works with 1.11.3. Change the scalatest line above to:

libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.0.1-SNAP" % "test"

Now reload sbt and the test code should compile cleanly.

Writing a Simple SBT Task

2013-12-17T12:41:00-05:00

In a project I'm working on, I wanted to copy a fat jar to a remote machine for testing.

The sbt-assembly plugin is handy for making a fat jar, but I didn't find any convenient plugins for just copying the jar somewhere. (SBT's built-in publish mechanism is overkill for what I need.)

It turns out that this is an almost trivial task to write. Unfortunately, figuring out exactly how to tie the pieces together isn't exactly straightforward.

I found this set of slides, while not comprehensive, to be about 100x more enlightening than the official sbt documentation. Starting with this slide especially, I "got" how the task could fall together.

The main sticking point for me was trying to figure out how to get the jar filename in my task. As the slides point out, operators that begin with < create a dependency on other values. Specifically, <<=:

declares a dependency and passes the right-hand value or method result to the setting represented by the left-hand key

I started from scratch, not worrying about the jar filename yet, just creating a stub task. The canonical "hello world" task is easily accomplished by adding two lines to build.sbt:

val helloTask = TaskKey[Unit]("hello", "Print hello")

helloTask := println("hello world")

(Note that I'm using SBT 0.13.)

Take note that := in the definition of helloTask means that it's not dependent on any other settings. If we redefine it as follows, it will depend on the assembly task, and the assembly task's value (a java.io.File) will be available in the hello task:

val helloTask = TaskKey[Unit]("hello", "Print hello")

helloTask <<= assembly map { (asm) => println(s"hello ${asm.getName}") }

Now when you run hello at the sbt prompt, it will run all the tasks required to generate the assembly jar, and then print the name of the jar in the hello message.

Armed with this (still shallow!) knowledge, and cribbing from this (obsolete) recipe to accomplish the same goal with an older SBT, let's get dangerous and add this task to build.sbt:

val deployTask = TaskKey[Unit]("deploy", "Copies assembly jar to remote location")

deployTask <<= assembly map { (asm) =>
  val account = "user@example.com" // FIXME!
  val local = asm.getPath
  val remote = account + ":" + "/tmp/" + asm.getName
  println(s"Copying: $local -> $account:$remote")
  Seq("scp", local, remote) !!
}

Enjoying Scala's Compactness

2013-12-11T19:55:00-05:00

As I slowly learn my way around the Scala APIs, I'm enjoying the expressiveness that is possible. You can do a lot with just a little bit of code.

Operating on Files

For example, I needed a way to perform an operation on every file in a directory tree.

/**
 * Call proc(f) for each file in the directory tree rooted at dir.
 *
 * proc will not be called for directories, just for files.
 */
def traverse(dir: File, proc: File => Unit): Unit = {
    dir.listFiles foreach { f => if(f.isDirectory) traverse(f, proc)
                                 else proc(f) }
}

That File is just a java.io.File, and its listFiles method returns an Array of File.

Scala's foreach calls the given anonymous function for each element in the array.

Unit is Scala's name for void.

Operating on Strings, Experimenting in the REPL

Related to files, but operating on strings, I needed to take a file, say content/topic/subtopic/post.md and generate the name where the output would land, in this case output/topic/subtopic/post.html.

This means splitting the filename by "/", dropping the first directory, prepending the output directory, and changing the expression from ".md" to ".html".

I had to experiment in the REPL (along with heavy use of ddg) to figure this one out, but it ends up being a (long) one-liner.

scala> val f = new java.io.File("content/topic/subtopic/post.md")
f: java.io.File = content/topic/subtopic/post.md

First, the File object gives us the parent directory of the file, and splitting on the separator gives an array of Strings representing the path.

scala> f.getParent split java.io.File.separator
res37: Array[java.lang.String] = Array(content, topic, subtopic)

I want to drop the first directory component:

scala> f.getParent split java.io.File.separator drop 1
res38: Array[java.lang.String] = Array(topic, subtopic)

and put the output directory at the front of the list:

scala> val out = new java.io.File("output")
out: java.io.File = output

scala> out.getPath :: ((f.getParent split java.io.File.separator).toList drop 1)
res42: List[java.lang.String] = List(output, topic, subtopic)

[If you're paying close attention to the REPL output, you'll notice that I'm leaving out my mistakes, i.e. res39 through res41.]

I had to convert the Array returned by split to a list, drop the first element of the list, and then prepend the output directory using Scala List :: function.

Now I can join the strings in the list together with a slash to get the output directory that the file belongs in:

scala> out.getPath :: ((f.getParent split java.io.File.separator).toList drop 1) mkString java.io.File.separator
res44: String = output/topic/subtopic

That java.io.File is kind of a hassle to keep typing out:

scala> import java.io.File
import java.io.File

scala> out.getPath :: ((f.getParent split File.separator).toList drop 1) mkString File.separator
res45: String = output/topic/subtopic

That's a little better.

Now let's take care of the filename, which is a trivial replacement:

scala> f.getName.replaceAll(".md", ".html")
res46: java.lang.String = post.html

and append it to the path we just generated:

cala> out.getPath :: ((f.getParent split File.separator).toList drop 1) mkString File.separator + f.getName.replaceAll(".md", ".html")
res47: String = output/post.htmltopic/post.htmlsubtopic

Whoops, that's not right -- it needs parentheses so that the mkString happens first, and an extra path separator:

scala> (out.getPath :: ((f.getParent split File.separator).toList drop 1) mkString File.separator) + File.separator + f.getName.replaceAll(".md", ".html")
res49: java.lang.String = output/topic/subtopic/post.html

So the function that I added to my class is (note that the out from above is called outDir here):

def outputName(f: File): String = {
    val sep = File.separator
    (outDir.getPath ::
        ((f.getParent split sep).toList drop 1) mkString sep) +
        sep + f.getName.replaceAll(".md", ".html")
}

Doesn't exactly look like a one-liner, but it could be if I didn't break it up for readability.

Building a Single File Scala Executable

2013-12-07T23:37:00-05:00

I'm from a C background with a hefty dose of Python, mostly on linux systems.

My preferred mode of delivering software is as a single executable file. I hate having to explain to users that they have to install a half-dozen support libraries. (This is less painful on modern linux systems where I can tell them to do the equivalent of apt-get install library1 library2 ... or just ship a deb/rpm/etc that explicitly contains the requirements.)

This doesn't seem to be the normal way of doing things in the Java world. (It's often not that simple in Python either, so I accept that I will have to make adjustments.)

The first Scala project I'm working on is a simple static site generator -- essentially to replace my use of Pelican with my own program in Scala. It turns out this is very easy to do with Laika, which looks like it will do most of the work for me.

So my first challenge was producing a file that I could actually run like a normal system command, without messing around with a long java command line, classpaths, etc. And without having to tell users that they have to install Laika or whatever other dependencies I end up with. This is fairly easy with sbt-assembly; the following instructions are largely copied from the sbt-assembly README.

Create $root/project/assembly.sbt with the following content:
```
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.10.1")
```
Add import AssemblyKeys._ at the top of $root/build.sbt.
Add mainClass in assembly := Some("my.main.Class") at the bottom of $root/build.sbt.
Create $root/assembly.sbt with the following content:
```
import AssemblyKeys._

assemblySettings
```
Run sbt assembly and get a jar file in target/scala-2.10/my-assembly-1.0.jar
Run the file with java -jar target/scala-2.10/my-assembly-1.0.jar.

I tried to take it another step further and make the jar file self-executable by prepending a shell script. This is possible with zip files (jars are just zip files) because the important info is at the end of the archive and anything can be prepended. But apparently java is stricter about the format because this doesn't work -- java complains "Invalid or corrupt jarfile".

I'll have to settle for a (two-line) wrapper script. I consider it a success that I was able to take the single jar from above, copy it to a machine that didn't have Scala or Laika installed, and it ran just fine.

Getting Set up for Programming in Scala on Debian

2013-12-06T21:01:00-05:00

Things we need to be able to use Scala on Debian (everything below will likely work on Ubuntu as well):

Scala itself, of course. apt-get install scala will pull in a bunch of packages -- all the java stuff, etc. However, note that sbt will happily pull down whatever version of scala you specify in your build.sbt. While this step may not really be needed, I'm assuming that it at least ensures that scala's java-related requirements are met. For what it's worth, debian is currently shipping scala 2.9.2 in "testing" (jessie), and I'm targeting 2.10, so my builds aren't using the system scala libraries.
sbt (simple build tool) is Scala's equivalent to make/Ant/Maven. Debian doesn't provide a package for sbt, but the sbt project produces a deb as part of the official release. Install curl first if you don't already have it, since sbt depends on it, and check the download page to make sure you're getting the latest version of sbt:
```
$ sudo apt-get install curl
$ wget http://repo.scala-sbt.org/scalasbt/sbt-native-packages/org/scala-sbt/sbt/0.13.0/sbt.deb
$ sudo dpkg -i sbt.deb
```
I'm planning on doing some work with Lift, and deploying via .war, so I need tomcat. On one site I'm going to be behind apache, so I'll set up for testing that way:
```
$ sudo apt-get install tomcat7 apache2
```

That's all for now. I'll come back and update this if I disover other setup items I'd have rather had at the beginning.

What Makes a Nice Programming Language

2013-12-05T19:15:00-05:00

(I started this post about halfway through the Year of New Programming Languages, and wrapped it up in December.)

At the midway point I was most excited by Common Lisp, but with doubts I'd be using it for anything real. By the end of the year my favorite new language was probably D.

Icon was fun, but impractical: no real libraries, no ecosystem, no way to deploy anything, no tools, etc. It's dead. Nothing more than a historical interest.

I don't see what the big deal is about Go. To be fair, I was trying to do a lot of string processing, which seems to be cumbersome in Go. There are probably other areas where it shines.

Scala is neat, but it's big -- it needs much more than a month to be able to do anything worthwhile with it, and you have to learn your way around the Java ecosystem if you aren't already familiar with it. And in the end, you're on the Java ecosystem, which... well, it is what it is, you take the bad with the good. (I'm prejudiced against it.)

It's hard to get excited about Java, which I'm only using to make a little game for my Android tablet. I'm trying to keep an open mind and fight the prejudice. On the bright side, there are lots of tools, which brings me to the point of this post.

Your language can suck (at least somewhat), but if it has a lot of high quality tools, it might be more usable than a theoretically superior language with crappy tools. In my opinion, this is Java: I don't care much for writing Java, but it doesn't completely suck and those guys have some really nice tools.
My goal when I start a programming project is to deliver some value to someone. Frequently, this is ultra-short-term value for myself (e.g. a one-off bash script). I also hack a lot on longer-term projects for personal use or by groups which I have a relationship with (e.g. small Flask/Python website). If I can't deliver the value, then what is the point?

I'm looking at you, Common Lisp: I couldn't figure out a way to reliably deliver a simple desktop app. Icon is in the same boat. On the other side of the coin, Java and Python can be really easy to deliver -- there are tools to generate desktop installers, and deploying web apps can be really simple. In the old days of the web, this was the beauty of Perl CGI scripts: to deliver it, you just drop myscript.pl into your cgi-bin. (I figure it's also why PHP, which is a crime against humanity, is so popular: deployment is trivial on almost any web host.) And Javascript has historically been more or less the only obvious choice if you want something to run in a browser, though with Dart and some of the new languages that compile to JS we can see that changing.

Languages like D or Chicken Scheme that compile to native binaries are easy to ship as well. (They're somewhat harder to 'ship' if you're a FOSS project and the 'product' is your code, only for the fact that your users are less likely to have the necessary compilers and libraries available.)

So normally the first question I ask myself is, "How am I going to ship this?" That really drives the choice of language.
Are there tons of libraries? If I can grab libraries that do 80% of the work my project needs, it makes me strongly consider using the language. This is a huge strength of Perl and Python: there are libraries for everything.

In today's world, just having libraries isn't even enough: they have to be discoverable. The modern languages with which I'm familiar all have some kind of package discovery and download tool: Perl has CPAN, Python has pip, Dart has a tool that I only interacted with through the IDE, etc.
What is the testing experience like? Does the language have a way (preferable one standard way) to set up and run tests? Does it not suck?

D has perhaps the best language support I've seen for testing -- you can declare unittest { ... } blocks right inline with the functions you're testing.

My limited experience with Perl gave me a taste of that ecosystem's good handling of testing. There are established patterns for how to do it The Right Way, and useful tooling set up around those patterns.

I'm most familiar with Python, where there isn't really a Right Way, but there is a unittest module in the standard library, and some popular tools like Nose that make unit testing not painful.

C on the other hand, is annoying for unit testing. There are libraries, but the quality varies, and mostly my experience has been that you have to roll your own support for testing, pretty much every time.
Programs written in the language need to be debuggable. Especially when I'm learning the language, because I'm going to do stupid things. If debugging is painful (or worse, if I can't even figure out how to get past compiler error messages -- I'm looking at you, ghc), then the frustration just makes me want to walk away and find a better tool.

Part of debuggability is making it so that I can seamlessly walk through library code. The Dart IDE provided a nice experience here. (Yes, I know it's Eclipse, but amazingly it didn't suck like Eclipse usually does. Not a single crash.) When I did dumb things in my code, I got a backtrace from the library code where I could see the incorrect argument I was passing down.

New Language Wrapup

2013-12-04T14:16:00-05:00

Learning a new programming language each month this year was an ambitious goal. I didn't quite make it -- other things got in the way. I did manage to dip my toes into some interesting languages, learn some new things, and find a couple of candidates for diving deeper into.

Here's the final list of languages I learned and mostly-incomplete projects/prototypes built with each:

Icon - markdown processor
Go - markdown processor
Common Lisp - make-like build tool
Scala - personal finance app
D - MP3 ID3 tag reader
Scheme (specifically, Chicken Scheme) - photo sharing website
Dart - web game

I played with Java, Groovy, and Rust a little bit, but didn't really do anything with them. (Java isn't new to me, though I'm far from "experienced" with it either.)

My favorite of the bunch has to be D, though I did enjoy working with Dart and Scheme.

In the end, though, D seems too familiar -- too similar to the things I know, and Scheme/Lisp aren't really new to me.

I debated between Scala and Haskell (which I played with a little bit last year), and thought about Erlang (which I experimented with a couple of years ago) and decided to make a deep dive into Scala over the next year. Scala offers a paradigm shift like Haskell or Erlang would, it also forces me to learn a new platform and ecosystem, and challenges my decade-plus JVM-hate. Maybe I'll even learn how to be effective with Eclipse.

Maybe not.

I hope to record my progress here as I go along, but my track record on hopes-of-more-frequent-blogging isn't great.

Recovering Photos From a Bad Hard Drive

2013-05-03T19:50:00-04:00

I've had a couple of friends ask me if there's any way I could read family photos off a dying hard drive. Obviously, the best way to do this is to have a backup: then you can throw out the hard drive (use an electronics recycler) and restore your backup to a new drive.

But if you've arrived at this page, you probably don't have a backup...

I'm writing this down mainly because I forgot the details after I did this the first time, and I don't want to have to do the research again.

This will only work if the drive is still alive enough to be mostly readable. If it is well and truly dead, I can't help you. Even if it's only half-dead, I still can't help you -- this is what worked for me, there are probably better ways to accomplish this, I'm not an expert in this area. (In other words: don't ask me for help.)

Connect the bad drive to a computer running Ubuntu. (Any linux will do, but some of the details below assume Ubuntu/Debian.)
apt-get install gddrescue testdisk (Note that "ddrescue" is not the same thing.) Gddrescue provides a "ddrescue" program, and testdisk provides "photorec".
Look in dmesg to see where the new drive lives. Mine is at /dev/sdc
Run sudo fdisk -l /dev/sdc and verify that the size and type are what you expect. In my case, the drive to be rescued is from a Windows box and formatted NTFS, so I know that I'm not mixing it up with any of the ext3 drives that are on my system.
Create a directory to hold the rescue image, logfile, and files that will be extracted from the image. I put mine in ~/HDD_Rescues/Friend1/. Obviously, the destination directory needs to be big enough to hold the contents of the drive to be rescued.
cd to that directory and run ddrescue /dev/sdc hdimage logfile. This will try to read from the entire disk, saving the result in hdimage, and keeping track of what has been done in logfile. The advantage of the logfile is that, if an error occurs or you need to interrupt the recovery, it can continue where it left off. One drive I recovered was broken enough that it would die after a few 10s of MB -- and needed a reboot to recover. So the pattern is: run ddrescue, wait for error to occur, reboot, repeat until enough data has been recovered.
Run photorec /d recovered_files hdimage and walk through the prompts. Recovered files will be dropped in a series of directories named recovered_files.NN, where NN is an incrementing number.

Generic Requirements for Build Scripts

2013-02-12T19:08:00-05:00

A few years ago, I wrote Makefiles are Software Too, which laid out some "generic requirements" for any build scripts (Make, Ant, Scons, Fabric, etc) that you're writing. I still think those are important, but over the past (nearly) four years I've worked with more systems, and I think there are some additional requirements we can add to the list.

As motivation for this list, consider the consequences of skipping the requirements below:

A build that simply rebuilds everything, every time will waste a lot of developer time. It's almost always a good investment to set up a system that will only rebuild what is needed; the developer time spent investing and maintaining such a system will be repaid through faster builds and less time spent waiting.
A build that doesn't rebuild everything, but does not do a good job of tracking dependencies and often fails to rebuild a component that should have been rebuilt results in strange bugs during development. Developers who work with such a system will be in the habit of doing make clean and you end up in situation #1.
If you're in the horrible situation where make clean doesn't even work, developers will be in the habit of nuking their tree and pulling down a fresh source tree. This takes even more time and carries the risk of deleting work!

A Single Command Builds Everything

To build the "normal" product(s), it should not be necessary to do make setup; make the_product; make installer. If you find yourself running series of commands like this, you probably need either a higher-level target (make all) or your bundle target should depend on the_product (and one or both of those targets should probably depend on setup).

Environment Variables are Encapsulated

The build script must encapsulate all environment variables within the script. (This is a corollary of the "single command" requirement above.) When writing build scripts, environment variables seem like a convenient way to add flexibility (I'm guilty of this), but in the end they add complexity for users of the script. I think it's ok if there are settings that can be customized for developers who need to do something different, but the script should provide reasonable defaults that "just work" for most of the developers (and the CI system).

What's even worse than environment variables required by the script is if there are parts of the toolchain that require variables that aren't set to a reasonable default by the script.

Rebuilding is Minimized

The build script should not unnecessarily rebuild source code in a directory tree that has previously been built. (This is make's raison d'etre.)

I have worked on more than one system at past employers and clients with make targets that include a long chain of commands to rebuild that target, where there are intermediate build products created along the way that could have been separate targets. Since those intermediate targets often would not need to be rebuilt every time, build times could have been reduced.

Documentation Exists

The build script must be documented. If anything, the scripts I've seen in the past few years are become more complex than those I saw in the previous decade. In "Makefiles are Software Too", I said:

It doesn't have to be elaborate, but a five-line comment at the top of the script describing the available command-line variables would be nice.

A pattern I've seen that's even better is to provide a make help command or the equivalent. This should echo the targets that are available for the user and what they do, as well as any variables that are commonly used to customize the build (e.g. DEBUG=1 for debug mode).

Build Trees are Cleanable

I actually think this is less important -- with two important assumptions:

The version control system provides a way to get back to a clean tree (e.g. git clean -f). (Though this can have the danger of removing new files that haven't yet been added to VC.)
Dependencies work reliably so that cleaning the tree is never needed.

If both of those requirements are met, then make clean is optional. But if such a target is provided, then it must work reliably. And if there are multiple levels of "clean" (e.g. GNU specifies clean, distclean, mostlyclean, and maintainer-clean), the behavior of each of these should be specified in the documentation.

Cleaning Should Not be Needed

This is mentioned above, but it's really a separate generic requirement. Cleaning (i.e. running make clean, git clean -f, or the equivalent) should be a rare event. If you find yourself doing this, your build script is defective -- there's likely a problem with dependencies, see below.

Parallelizable

I originally listed this as a "bonus", but I think that this is more important now than it was in the last decade. Multicore systems are the default now, so the ability to spread a build across all the processing power available on the system is necessary. If all of the other generic requirements for build scripts are satisfied, it shouldn't be difficult to run a parallelized build. (And if your tool doesn't support building in parallel, you need to find a new tool!)

This also supports spreading large builds out over multiple machines, e.g. with distcc).

Dependencies are Automatic

Wherever possible, dependencies are handled automatically. This means that, for example, your Makefile automatically knows that foo.o gets built from foo.c, and it automatically generates a list of all of the header files that foo.c includes. This is almost trivially easy to do with GCC's -MD flag.

Note that, if I have to run the equivalent of make depend, then dependency generation is (by definition) not "automatic". Dependencies must be generated on the fly and updated whenever anything changes.

Don't Implicitly Touch VC

In my opinion, the build system should not touch version control at all, but I accept that this is just my opinion. I've seen several builds that are based on fairly complex VC configurations, and having support for managing multiple-repository builds as part of the build scripts can be a useful feature.

As a requirement though, I think it is reasonable to say that the build system should not touch VC (neither getting updates nor putting changes) as part of the default target path. Any build commands that affect VC should be obviously documented as doing so. (It's dangerous to commit changes before you're ready. It's even more dangerous in some VC systems to pull down/merge into your workspace before you're ready. Yes, there are still people out there using CVS...)

Build Targets are Granular

This is complementary to the "Single Command" requirement. While we want to enable users to be able to build the whole product all at once, we also want to enable users to rebuild specific parts and avoid having the overhead of creating products they don't care about.

Consider the case where you've got a very large product, but your immediate task is to fix a bug in a smaller component. You've written a unit test that exposes the bug and/or the component can be run standalone for testing. While you're in an edit/compile/debug loop, you don't want the overhead of rebuilding the whole product, you just want to rebuild your component and the tests that go along with it.

The build script must provide targets so that users can rebuild specific components instead of the entire product.

Run Tests

The build script should support running unit or other tests. (E.g. GNU's make check.) Where applicable, tests must be rebuilt automatically before being run (this is a corollary of "Dependencies are Automatic").

The reason this belongs in the build script is that unit tests are part of the depdendency chain, and we mustn't be required to manually determine when unit tests need to be rebuilt. Whether to include other sorts of tests under the bailiwick of the build script is a decision for the implementor.

It would be nice to be able to run specific tests, but it doesn't seem like this needs to be provided through the build script.

Gebildet: Writing Ebooks in Markdown

2013-02-03T09:24:00-05:00

Last month I wrote a prototype markdown processor as part of learning the Icon programming language. This month I'm rewriting it as part of learning Go, but this time with the intention of maintaining the tool over the long term.

As mentioned in the previous article, "gebildet" is intended to be a better dexy. I want to be able to write a book (which contains a lot of source code) in Markdown, but keep the code files separate. The tool should allow:

inserting the code into the final product,
compiling and/or running the code to verify that it is correct, optionally capturing command/code output into the final product,
inserting the output from arbitrary commands into the final product,
extending the tool with additional "directives" (a Markdown syntax extension).

Gebildet will not generate a "final" product (i.e. PDF or ePub) -- this is the domain of a separate tool. What it will generate is "standard" Markdown that can be consumed by a tool like pandoc to generate a final product in whatever format is desired.

Testing HTTP Basic Auth in Flask

2013-01-30T23:29:00-05:00

I created a quick & dirty admin page for a flask-based website I'm working on, and I had to add HTTP authentication support for it. Doing this is easy.

In the app, I add:

import auth

# ...

@app.route('/admin/', methods=['GET'])
@auth.requires_auth
def admin_page():
    # ... code to fetch info from db ...
    return render_template('admin.html',
        # ... args ...
        )

Note the @auth.requires_auth decorator. This uses an auth module (code stolen from this snippet):

from functools import wraps
from flask import request, Response

def check_auth(username, password):
    """This function is called to check if a username /
    password combination is valid.
    """
    return username == 'me' and password == 'my_password'

def authenticate():
    """Sends a 401 response that enables basic auth"""
    return Response(
    'Could not verify your access level for that URL.\n'
    'You have to login with proper credentials', 401,
    {'WWW-Authenticate': 'Basic realm="Login Required"'})

def requires_auth(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        auth = request.authorization
        if not auth or not check_auth(auth.username, auth.password):
            return authenticate()
        return f(*args, **kwargs)
    return decorated

Easy enough to do a one-off test, but it really ought to have an automated test. Unfortunately, I don't see support for setting headers in flask's built-in test client or in the Flask-Testing extension. That's ok, we can use the test client in Werkzeug.

First, the imports we need:

from flask.ext.testing import TestCase
from werkzeug.test import Client
from werkzeug.datastructures import Headers
import myapp

Note that the TestCase class we're using requires Flask-Testing (pip install Flask-Testing; I'm using version 0.4). This test class wants us to set up the app:

class AuthTest(TestCase):

    def create_app(self):
        views.DATABASE = 'testing.sqlite3'
        if os.path.exists(views.DATABASE):
            os.unlink(views.DATABASE)
        query_db(self.CREATE_TABLE)
        return myapp.app

Then writing a test case to verify that the admin page is locked is straightforward:

    def test_admin_page_is_locked(self):
        rv = self.client.get('/admin/')
        self.assert_401(rv)
        self.assertTrue('WWW-Authenticate' in rv.headers)
        self.assertTrue('Basic' in rv.headers['WWW-Authenticate'])

But actually sending an Authorization header for the client is a bit more challenging. Note that self.client is a subclass of Werkzeug's test.Client, and the open method there supports passing in extra Werkzeug Header objects. This is ugly, but it works. The first two cases below test login attempts with bad password and username, respectively, and the final test case logs in with the correct credentials.

    def test_admin_page_rejects_bad_password(self):
        h = Headers()
        h.add('Authorization', 'Basic ' + base64.b64encode('me:foo'))
        rv = Client.open(self.client, path='/admin/',
                         headers=h)
        self.assert_401(rv)

    def test_admin_page_rejects_bad_username(self):
        h = Headers()
        h.add('Authorization',
              'Basic ' + base64.b64encode('foo:my_password))
        rv = Client.open(self.client, path='/admin/',
                         headers=h)
        self.assert_401(rv)

    def test_admin_page_allows_valid_login(self):
        h = Headers()
        h.add('Authorization',
              'Basic ' + base64.b64encode(me:my_password))
        rv = Client.open(self.client, path='/admin/',
                         headers=h)
        self.assert_200(rv)

If I have some time when I'm done with this project, I'll probably send a patch to Flask-Testing to add support for headers= to that test client.

A Small Project in Icon

2013-01-26T16:30:00-05:00

(Background: 12 Projects in 2013)

I wrote a bit about the general experience of programming in Icon a couple of weeks ago. This post is a brief post-mortem about a project that I tried to build using Icon. I'm not planning on releasing the source, because I don't consider the results very useful except as a personal learning experience about the Icon language and the problem domain. (And because I plan to rewrite the software in next month's new language: Go.)

The goal is to parse a book written in slightly-augmented Markdown into either HTML or "standard" Markdown so that other tools that consume Markdown can be used to generate friendly formats like PDF or ePub.

I was inspired by dexy, which works reasonably well, but I found it cumbersome to use.

The limitations in standard Markdown that I intend to augment are the ability to include files, and to include the output of shell commands. I added support for the syntax !!directive args...; where the processing tool defines a handful of directives, e.g. include, bash_interactive, etc. These could be used, as in dexy, for ensuring that code samples are correct, and that the output from these samples is properly reflected in the text of the book.

As far as doing this in Icon goes: the language's string-handling facilities made parsing Markdown easy. The problem I ran into that caused the most difficulty is that Icon doesn't provide a good mechanism for grabbing output from subprocesses. The open() function takes a "p" flag for reading/writing from/to a subprocess, but it's nowhere near as rich as Perl, or, say, Python's subprocess module. For one thing, there's no built-in support for capturing both stdout and stderr. Also, it isn't obvious how to check the exit status of a subprocess spawned via open().

If I was committed to using Icon for this project, I could have written C functions to provide better subprocess/pipe support, but the end of the month is approaching and I think it makes more sense to treat this as a prototype and rewrite it in Go (a language with a strong future) next month.

Programming in Icon

2013-01-14T20:11:00-05:00

(Background: 12 New Languages in 2013)

Icon is a cool little programming language. It's worth learning, even if it isn't practical for daily use.

Pros:

Syntax is similar to C, which makes it easy to learn.
A handful of publication-quality Icon books (now out of print) are available for free download as PDF. I used The Icon Programming Language, 3rd Ed as my primary resource.
Latest version was released in 2010 and has been validated against newer systems (Ubuntu 12.04, Mac OS 10.7).
"Everything is an expression", expression failure, backtracking, and some of the string functions are different enough to make your brain exercise new pathways when solving various problems with Icon. (This is one of my benchmarks about whether it's worth learning a new language.)
There are debian packages (apt-get install icon-ipl iconc icont iconx).

Cons (some of these are just nits, really):

Documentation (at least what I found online) for the library is poor (at best).
1-based indexing; this has caused me a handful of hard-to-find bugs. ("Positions in Icon" is kind of odd, too. It's not what you're used to.)
It's basically a dead language. Newest version was released in 2010. The FAQ says:

We continue to use Icon on a daily basis, but no significant changes are planned.
Runtime error reporting is dreadful. Better with the interpreter, but more or less unusable with the compiler. Not much is caught at compile time.
Do you known any Icon fanboys? Any popular projects that use Icon? Yeah...

On balance, it's not something I'd reach for to solve a given problem, but I'm writing a book processor with it (more in a future post), and I'm finding that it's up to the job of mangling text.

12 Little Projects in 2013

2013-01-14T19:08:00-05:00

As I mentioned in 12 New Languages, I'm learning a new programming language each month in 2013. To go along with this effort, I need a dozen things worth building -- one for each language.

I'm going to try to play to the strengths of each language. Here's the initial lineup:

Icon: "bucher" -- write books containing tons of code, using Markdown
Go: "gebildet" -- better than bucher
Common Lisp: yet another "make" tool; rogue-like game
Scala: finance app with a Swing UI
Java: a simple word game for Android
Dart: rogue in "html5"
D: ?
Ruby: ?
Groovy: scraping git (via jgit) for more-than-stats (finding buggy modules?)
Fortran: ?
Squirrel: ?
Forth: ?

(Scheduled projects subject to change at my whim...)

12 New Languages in 2013

2013-01-05T18:26:00-05:00

I'm setting a personal goal in 2013 to learn a new programming language each month. Obviously, I won't be able to learn any of them in depth, but I plan on completing a small project in each language.

January: Icon
February: Go
March: Common Lisp
April: Scala
May: Java (oh the horror!)
June: Dart
July: D
August: Ruby
September: Groovy
October: Fortran
November: Squirrel
December: Forth

(Scheduled languages subject to change at my whim...)

Edit: 26-Jan, swapped Groovy and Go in February.

Edit: 18-Feb, drop Clojure for Common Lisp, swap Groovy with Lisp in March.

Edit: 1-May, drop Pike for Java (to enable an Android project), move Scala to April.

Get notification when long running commands are done

2012-06-22T18:44:00-04:00

I often launch a long running command (e.g. a build or firmware upgrade), then wander off to another virtual desktop to take care of a few small tasks while I'm waiting for the command to complete. Unfortunately those "little tasks" tend to suck me in and I forget that I was just filling in time while waiting for something.

Adding the function below to my zshrc makes it so that I can do things like lr make check and have a dialog pop up in my face when it's done -- even when I'm ssh'd to a VM or a remote machine. (Assuming I've used ssh -X remote.)

function lr()
{
    $*
    if [[ $? == 0 ]]
    then
      success=succeeded
    else
      success=failed
    fi
    zenity --info --text "command $success: $1"
}

Offline RSS Readers on Linux

2012-06-22T07:24:00-04:00

Building an RSS reader must be some kind of rite of passage. Everybody has probably started one at one point. (Hey, I've done it too.) It seems easy enough, right? Unfortunately, it is pretty easy to get something sort-of working. Which means that there's a lot of junk available. And it's a double-edged sword that Ubuntu has quality bar of virtually zero: tons of packages very easily available, but YMMV.

What follows is a quick rundown of the available RSS readers in Ubuntu, so that you, dear reader, don't have to experiment with all the same junk I did. (For what it's worth, I was running Ubuntu 10.04 LTS when I did this review, so it's possible that some of these projects have improved in the last 20 months. At some point I will update this with results for Ubuntu 12.04 LTS.)

My evaluation is relatively shallow. I'm installing the package via apt, running it (from the console, so I see whatever turds get printed to the console), importing three OPML files from Google Reader, and then reading items in a handful of feeds. I also attempt to add a couple of feeds manually, including at least one podcast.

blam: JUNK (1.8.7-1ubuntu1)
- Spits out exceptions on the console when, for example, name resolution fails. This happened during import of my feeds.
- About half of my feeds won't update -- no posts are displayed for the feed. Refreshing manually doesn't help. The URLs are valid and there should be posts.
- Removing the example feeds is annoying. I can't just push the delete key while a feed is highlighted, and there's no keyboard shortcut. If you use keys to navigate the menu, three of the menu items use "R" as the accelerator key.
- In one of the feeds that does have posts, clicking on any post to read it causes blam to open a Nautilus window on /usr/share/themes/blam. WTF? I stopped here -- this application is clearly unusable.
liferea: GOOD (1.6.2-1ubuntu6.1)
- Removing the example feeds is slightly less annoying than blam. Delete key doesn't work, but at least the menu uses a single accelerator key. The confirmation dialog is annoying because "Cancel" is the default action instead of "Delete". A minor nit: they mix usage of "Remove" and "Delete".
- Double-clicking the feed doesn't bring up an "Edit Properties" dialog -- you have to right-click to get there. At least blam got this right.
- Fails to update if you provide an URL that serves a webpage with a link rel=alternate tag specifying an itpc: URL, even though the second link rel=alternate is an http: URL. Manually looking at the webpage source and specifying the correct feed URL does the trick.
- When adding a new feed URL, you aren't presented with the box to "automatically download enclosures". You first have to add the feed and then edit properties to tick this checkbox. This means that the first download won't fetch the enclosures.
- A podcast feed that includes the same URL as both an "enclosure" and "media:content" causes liferea to spawn two instances of wget, each downloading the same URL into the same local file. The media file is also shown twice at the bottom of the post, though this is minor.
- Double-clicking the downloaded enclosure in the post window prompts to spawn a program to play the MP3 file. Given that liferea is a GNOME-aware application, it seems reasonable that it should use my preferred application for MP3s, just as if I had double-clicked the file in Nautilus.
- When I specify /usr/bin/mplayer as the application for MP3s, there's a long delay -- possibly because the file is still downloading? I double-clicked it again, thinking that I might have mis-clicked. When it finally started, there were two players playing the same file, and no apparent means of killing it through liferea. (Indeed, when I look at the process list, I can see that mplayer doesn't show up as a child of liferea.) Another nit: when liferea spawns the player, it doesn't send stdout/stderr to /dev/null. This means that mplayer spews all kind of noise onto the console. (Turns out that totem is a much better choice -- so that you have a GUI to pause/stop/kill.)
rssowl: (2.1.2-1\~getdeb1)
- Best installation experience so far. When starting up for the first time, I'm presented with a tutorial and a sort of import wizard instead of a lame list of "sample feeds" that I have to delete.
- The import process is the best seen so far. It displays the feeds from the OPML file, offers to flatten the hierarchy and asks where in the existing folder hierarchy to place the new feeds.
- Rssowl has the same "itpc: URL" issue mentioned for liferea above.
- Podcast support is interesting. When an enclosure is present, an icon is shown in the post. You can click the icon and it will offer to download the file or create a "filter". Creating a filter is like a GMail filter -- it will be performed on every post in the feed. In this case, the filter is automatically constructed to download the enclosures to my \~/Podcast directory. Then, in the window listing all filters, I have the option of running this new filter -- which downloads all of the enclosures.
- When the enclosure is downloaded, I can click on the icon in the post and am presented with "Open abc.mp3". Clicking this automatically opens totem -- the application that is registered with GNOME to play MP3 files.
- Doesn't leave any turds on the console when run from the command line. (To me, this is a sign of polish. Turds aren't really a "bug", but they just don't belong there.)
- Searching/filtering is far superior to liferea.
- Paging through articles was a little clunky, though this may have been operator error. On the plus side, there are reasonable keyboard shortcuts for the common things I tried -- next article, next unread article, next feed with unread news, etc.
- Odd behavior: whenever I switch virtual desktops, Rssowl minimizes itself to the tray. Really annoying when you flip back and forth between the app and the browser to write a review like this...
yarssr: Meh (0.2.2-5)
- This is a simple app that displays headlines in the tray.
- Didn't import OPML with nested folders.
- Locks up when updating -- no UI interaction permitted.
- Once updated, you can click to see headlines. Choosing a headline opens browser window on the post.
- Doesn't do what I need, but for what it is, it doesn't completely suck.
rssreader.app: DOA (0.6.2.dfsg-4build2)
- DOA: Probably because it's based on GNUstep and I'm not running GNUstep. I installed via apt and it pulled in several packages. When I tried to run it, I got a segfault. Didn't do anything further, just purged it from my system.
rawdog: Underpowered (2.12.dfsg.1-2)
- Except for the rssreader.app segfault, possibly the most braindead installation of anything. Running rawdog, you're greeted with "No /home/brian/.rawdog directory". Grr. If I weren't writing a review, I'd just uninstall it here and never look back.
- This is a batch / command line reader. You add feeds via command line, run an update command, and run a command to write all of the fetched posts into an HTML file. Then you can read the HTML file in your browser.
- Doesn't do what I need, but for what it is, it seems to do a good job.

Quick & Dirty “sniffer” for when you don’t have tcpdump

2012-06-08T19:55:00-04:00

I recently needed a way to test if packets were reaching a linux device I'm testing. The device didn't have tcpdump installed, so I didn't have a real sniffer. Fortunately all I really needed was confirmation that packets were arriving. iptables to the rescue! Assuming there are no existing rules to interfere (i.e. use iptables -F to clean out any existing firewall), you can use this to log UDP packets arriving at port 1234:

iptables -A INPUT -p udp --dport 1234 -j LOG

Then tail your syslog and you'll see a message when traffic arrives on UDP 1234.

Don't Forget dmenu

2012-05-18T19:03:00-04:00

When you install i3 on Ubuntu via apt-get install i3-wm, it should come with dmenu, but it doesn't.

Dmenu is a handy launcher that you can (by default) invoke with mod-d, then start typing the command you want to run (with dynamic completion) and press enter when the command you want is highlighted. But it won't work if you don't have dmenu installed, so be sure to do apt-get install dmenu.

Moving Through i3 Workspaces

2012-05-16T07:07:00-04:00

The default i3 config file only has commands to move to a specific numbered workspace (e.g. mod-1, mod-2). For better or worse, I'm in the habit of "traveling" through the workspace list using Ctrl-Alt-Up and Ctrl-Alt-Down (or Left and Right, if you prefer a horizontal feel for your workspaces). This is trivial to do in your ~/.i3/config:

# Use mod-Control-Up and Down to rotate through the workspace list.
bindsym $mod+Control+Up workspace prev
bindsym $mod+Control+Down workspace next

# Use mod-Shift-Control-Up and Down to move the selected window to an
# adjacent workspace. (This does *not* change your view to that workspace,
# it just "zaps" the window there.)
bindsym $mod+Shift+Control+Up move workspace prev
bindsym $mod+Shift+Control+Down move workspace next

Getting dnsmasq, virtual machines (libvirt), and vpn to inter-operate

2012-05-15T06:43:00-04:00

Fixing Dnsmasq to Work With Libvirt

There's a bug written in the Ubuntu tracker (and in Fedora too, I think) about running dnsmasq and libvirt at the same time. Basically, libvirt runs an instance of dnsmasq, and if you install the system dnsmasq it tries to bind to all interfaces by default. The "easy" workaround is to use the interface and bind-interfaces configuration options in dnsmasq.conf. On my machine, I've got:

interface=eth0
interface=br0
bind-interfaces

This makes it so that the system dnsmasq only listens on eth0 and br0, and does not bind to the wildcard address. Now you can run two instances of dnsmasq: the system instance, and libvirt's instance.

Getting The System Dnsmasq to Answer for the VMs

I wish I knew the answer for this: I'd like for my system dnsmasq to respond to queries for "fedora16" and "winxp" (hostnames of two VMs I use). I tried a couple of things that didn't work, and settled for a subpar solution.

First, I thought I might be able to use a fake domain ".vm" and tell the system dnsmasq to forward queries for that domain to libvirt's dnsmasq. This may be a workable solution; it didn't work when I tried it, but I was fighting other issues with libvirt at the time and I haven't had a chance to go back and try it again. The problem I ran into is that libvirt's dnsmasq was receiving queries for "fedora16.vm" but it didn't have ".vm" configured as a local domain. You can't monkey directly with libvirt's dnsmasq config (at least not that I saw), or this would be an easy fix. I tried changing the names assigned in the libvirt XML file dealing with DHCP names and addresses, but didn't get anywhere. Again, more experimentation here might win the day. (Definitely let me know if you know how to solve this.)

I also flailed around with some other config options trying to get those names forwarded to libvirt's dnamasq but couldn't come up with anything clean.

What I finally settled on is adding the names of the VMs to my local host file:

# Virtual Machines; this should match up with the DHCP assignments in
# /etc/libvirt/qemu/networks/default.xml
192.168.122.22 fedora16.home.bstpierre.org fedora16
192.168.122.23 winxp1.home.bstpierre.org winxp1
# etc...

This file is referenced by dnsmasq.conf via addn-hosts=/etc/dnsmasq.hosts. It's not perfect, since I have to remember to enter the addresses in two places (default.xml and dnsmasq.hosts) when I create a new VM, but I don't do that often so it won't be a huge burden.

Getting Names from the VPN

When I connect to $DAYJOB VPN, I need to be able resolve names for that domain from the corporate (internal) DNS servers. Dnsmasq has a config option that allows you to tie a certain domain to a certain set of DNS servers:

server=/example.com/172.18.1.1
server=/example.com/172.18.12.1
address=/vpnserver.example.com/169.254.123.210

The first two lines set the DNS servers when I'm on the VPN. The last line makes it so that I have the public address for the VPN server when I'm not yet connected -- since the domain matches, dnsmasq will try to forward the request to the internal DNS servers, which is guaranteed not to work. This is slightly clumsy since the address is hardcoded, but it doesn't change in practice and I can always query it directly from upstream DNS (e.g. dig vpnserver.example.com @192.168.1.1 to get it from my router).

Finally, I don't want to be making requests to those 172.18 addresses when I'm not on the VPN, so I set up this interface definition in /etc/network/interfaces:

iface tun0 inet manual
pre-up ip route delete 172.18.0.0/16 || true
pre-up service openvpn start || true
post-down service openvpn stop
post-down ip route add blackhole 172.18.0.0/16 || true

After the VPN is taken down, this will add a blackhole route for 172.18.0.0/16, which is where those DNS servers live. Just before the VPN is started, it removes the blackhole route so that I can get to those servers once the VPN is ready.

Controlling Audio with i3

2012-05-12T07:52:00-04:00

As I mentioned in the last post, I've switched to the i3 window manager. After fixing the Caps Lock key, the next order of business is getting the audio keys on my multimedia keyboard to control audio output. (It's nice to be able to quickly mute or pause when the phone rings, for example.)

This turns out to be pretty easy. I used xev to figure out the keysyms for the audio keys, and then added bindings to ~/.i3/config:

bindsym XF86AudioRaiseVolume exec pactl set-sink-volume 0 +5%
bindsym XF86AudioLowerVolume exec pactl set-sink-volume 0 -- -5%
bindsym XF86AudioMute exec ~/bin/toggle-mute
bindsym XF86AudioPlay exec clementine --play-pause
bindsym XF86AudioStop exec clementine --stop
bindsym XF86AudioPrev exec clementine --previous
bindsym XF86AudioNext exec clementine --next

As you can see, I prefer clementine as an audio player. If you have something different that you like, look up the command line method for controlling the player.

The pactl command controls pulse audio. The toggle script is something I threw together because pactl doesn't support toggling the mute, you have to tell it mute or unmute:

1
2
3

#!/bin/bash
toggle=$((pactl list sinks | grep -q Mute:.no && echo 1) || echo 0)
pactl set-sink-mute 0 $toggle

I only have one sink, so if you have more than one, you'll need to adjust the script.

Making Caps Lock an Additional Control in XWindows

2012-05-11T06:49:00-04:00

Background: I just built a new computer, and "upgraded" to Ubuntu 12.04. I've been a fan of Ubuntu for a long time (first Debian, then Ubuntu because of the faster / more predictable release cycle). But you can add me to the list of people unimpressed with the "Unity" desktop environment, and not terribly impressed by Gnome 3. Since I'm in the middle of an upgrade and learning about new features anyway, I figured I'd make the leap to a better window manager. I've settled on i3, which gives me a lot of power over how things are arranged. Of course it also means that I've got to configure a bunch of stuff the way I want it, more so than with the default Gnome/Unity/Metacity scheme. This potentially means some short-ish posts here about tweaks & tips for working with i3.

The first item of business is getting rid of Caps Lock. The example config in xmodmap(1) will swap your caps lock and control, which is definitely not what I want. The key to the left of the "A" (which happens to be labeled Caps Lock) should be treated the same as the left Control key. So I created ~/.xmodmap with the following contents:

!
! Make Caps_Lock an additional Control key
! (This is *DIFFERENT* than *swapping* Caps_Lock and Control, as described
! in the xmodmap(1) man page.)
!
keycode 66 = Control_L Control_L Control_L Control_L Control_L Control_L Control_L Control_L
clear lock
add Control = Control_L

Then, in your ~/.i3/config add:

exec_always xmodmap ~/.xmodmap

(Note that if you use the xmodmap commands provided in the xmodmap man page, then restarting i3 will toggle that key between Caps_Lock and Control, which is probably not what you want! It seems that you also lose the swap if you do things like run xev.)

GTK3 Example Code

2012-05-07T04:38:00-04:00

Over the past year or so I've been doing a fair bit of GUI programming -- both in ${DAYJOB}s and otherwise. Until just recently most of my GUI experience has been with wxWidgets (and most of that via wxPython). Lately I've wandered to GTK.

The documentation for both toolkits is pretty good. I've found GTK's API reference documentation to be very thorough and mostly well detailed. The tutorials are especially good for an experienced programmer who knows his way around a GUI and just needs to learn how the toolkit works.

What I find lacking are a comprehensive set of tightly focused example programs. Take the clipboard example from gtkmm, for instance. It's supposed to be showing how to access the clipboard, but it's all cluttered up with irrelevant buttons and signals and stuff. Sure, it's not that hard to read through the code to pick out the relevant pieces, but why not produce a simple example app that's all relevant (or nearly all -- you can't avoid some of the boilerplate)?

And some topics are not covered at all -- like accelerator groups, where the API documentation says "you won't usually need these". But sometimes you do, and then it's hard to find an example of correct usage. (Yes, I figured it out, and it didn't take too long using the API reference, but a 25-line example program would have saved me some time.)

Don't misunderstand the point of this post: I'm not complaining about the state of the documentation, which is better than the docs for 98.3% of FOSS projects. Instead I'm proposing a collection of ultra-focused example programs for the breadth of the API. This will be a complement to the tutorials and reference docs.

It will ideally be against multiple language bindings (though I've found that an example in the C API is informative on how things should work in the Python or C++ binding, and vice versa). I've already got a few examples (C only so far) in the GitHub repository, and if life doesn't get in the way, I'll be adding more on a regular basis. In my dreams, I'll post some of the examples here with more explanation, but we'll see how that goes...

If you'd like to contribute a tightly focused example on some GTK feature, send a pull request on GitHub. If you'd like to request an example, open an issue ticket there.

User Interface Hall of Fame

2012-05-05T21:38:00-04:00

A long, long time ago, I talked about a poor user interface. This is the follow-up "good user interface" post that I promised.

I'd like to talk about Anki, which is a flashcard program. The user interface is well done for several reasons:

It is uncluttered: in the main screen you are presented with the flash card decks that you have downloaded. You can open a deck (i.e. to study the deck) by clicking the button right next to the deck. The number of cards that you have "due" to study today is shown right there.
When you've opened a deck, you are presented with the "Study Options" screen. At this point, the main thing you want to do is review your cards. Again, this is a big "review" button, and it is selected/highlighted by default so you can just press enter/spacebar to move into review mode.
- (I will note that the Study Options screen might be overcomplicated. There are a number of options here, and some information presented that does not seem essential. I'd say it's a minor fault that does not detract overall from the simplicity of this application.)
When reviewing, you are presented with one "face" of a flashcard and a "Show Answer" button. Again the button is selected/highlighted by default so you can just press spacebar to see the answer.
Once the answer is revealed, you click a button to say if you got the answer correct, and how easy it was for you to know the answer. So, for example, you're studying a French->English deck, and the face of the card is "non". Well, that's an easy one, it's "no". So when asked, you press the Very Easy button. Or, in a nice interface touch, you press "4", since the buttons are mapped to the first four number keys: "Again" (meaning incorrect) is 1, "Good" is 2, "Easy" is 3, and "Very Easy" is 4.
- Because of the layout of the buttons, the UI can be driven very quickly and easily from the keyboard: see a card, say the answer, press space to show the answer with the thumb, press 1-4 with one of the fingers, and then the next card is shown.
Unobtrusive status information is shown in the bottom left statusbar. It shows the number of failed cards that you need to review, the number of cards awaiting review, and the number of new cards today. It also shows the amount of time that it expects you have remaining to study, based on your average time to answer each card and the number of cards remaining. And then there are small graphs that indicate by color and size whether you've done well (green, >80% correct) or poorly (red, <50% correct).

Anki also has a facility for editing cards, but I'm not going to review that. The primary study feature is really the key piece of the application, and it deserves a spot in the UI Hall of Fame.

User Interface Hall of Shame

2012-02-22T10:06:00-05:00

I'm taking MIT's 6.831: User Interface Design and Implementation through their free OpenCourseWare website. One of the homework assignments is to find two examples each for the "UI Hall of Shame" and the "UI Hall of Fame". This post is for shame -- and since the nominee has so many things to discuss I'm only doing one; I'll follow up with fame in another post.

My nominee for the UI Hall of Shame is gmusicbrowser, which has a number of UI problems. I used this app for a while last year because it has a couple of neat features that I liked, but I got frustrated by it (and it doesn't tie into my multimedia keyboard play/stop/forward keys), so I moved on to something else.

A minor point, but the "About" dialog is not on the Help menu, it's on "Main" (which is nonstandard; it's where you'd normally find a "File" menu).
There isn't a "Help" menu.
The two (three? four?) icons below the menu are confusing at best. None of the following behaviors are what you'd expect, and not everything is consistent.
- To the right of the icons is a pair of numbers, like "4972/6158". This represents the sequence number of the playing song and the number of total songs in the currently displayed playlist. Nothing wrong with this -- but we'll come back to this in a minute.
- The first icon can be clicked to toggle between in-order and weighted-random modes. By right-clicking you can get a context menu to adjust the in-order mode (e.g. Artist/Album or Path/Track#). The selection for weighted-random mode is on a submenu of that same context menu.
- When you select a mode, that menu item gets a little indicator dot. It might be my desktop color scheme, but white-on-gray is hard to see with a dot that small. Firefox, for example, uses a checkmark which is easier to see even though it's still the same low contrast.
- Choosing a weighted-random mode changes the icon to dice.
- You can choose "Shuffle" from the menu, in which case the icon changes to playing cards.
- Clicking (left-click) on the dice or cards icon changes it back to the arrow icon and sets the mode to an in-order mode.
- Clicking on the arrow icon changes it to whichever random/shuffle mode was previously in use.
- The second icon starts as a book. I'm not sure what this is supposed to mean. When you hover over it, it says "Playlist Filter: All". Again you can right click for a context menu of filters to apply to the playlist, in which case the icon changes to a funnel. (And if a song is playing, it will abruptly change to whatever is first in the selected filter.)
- Clicking (left-click) on the book icon doesn't seem to do anything.
- Clicking on the funnel changes it back to a book (and the "All" filter?).
- Not that it would make much sense, but for consistency I'd expect a left click on this icon to toggle between "All" and the last selected filter.
- Next to the book/funnel icon is a blank space... that becomes an arrow/triangle ("play button") icon when you hover, with the tooltip "Queue empty". And when you move your mouse off the icon, it disappears.
- If you enqueue a song, that queue icon appears. And the pair of numbers mentioned above is replaced by the text "1 song in queue". But the playlist doesn't change! If it's still got 50 songs, you're still listening to, say, 12/50.
- Clicking on the queue icon... empties the queue! Which is very much not what you'd expect. Of course, since the queue is empty, the icon disappears and we get back our "NNN/XXX" text mentioned in the first bullet above.
- Now let's say you've got 5000 songs in your collection, but you're listening to a playlist that has 10 songs. You see, say, 2/10 in the indicator, and you've got the funnel icon (not the book). Just for fun, you enqueue a song using the main browser window. After the current song is playing, your queued song will play -- and the funnel icon will switch to the book icon: your selected filter has been cancelled.
- There are more unpredictable behaviors buried in these icons, but what all of this boils down to is that there are too many modes, they are not orthogonal, it's too easy to accidentally change from one mode to another, it's not easy to know what mode you're about to switch into, and it's really easy to lose work -- by accidentally clearing a bunch of queued songs.
Just below those icons are a mostly standard set of back/stop/play/forward icons... and a volume control that is really hard to use. Clicking the volume icon pops up a slider that disappears as soon as you release the mouse button. If you hold down the mouse button, the slider stays displayed, but you can't move the slider thumb. Double clicking the icon (which I discovered accidentally) makes the slider stay, and then you can move the thumb -- as long as you don't accidentally move the pointer outside the slider, in which case it disappears again. Right clicking toggles mute, which isn't intuitive, but if you really want to tie a behavior to right clicking on that icon, I suppose is reasonable. For mute, however, it's totally unusable: that icon is too small of a target, and when I want to mute, I'm usually trying to do it somewhat quickly -- so I can answer a ringing phone, for example. The mute button on my "multimedia" keyboard is a much easier target to hit quickly.
The main browser window is a mystery to me. Starting at the top, there are two tabs: Library and Context. The only thing in the Context tab is a checkbox for "Follow Playing Song"; I'm not really sure what this does. Assuming it does something meaningful, it would make much more sense to put this on a menu somewhere.
The Library tab is where you can browse your music library. There are three listboxes at the top of the window, each with the same set of tabs. Those tabs change the listbox contents so you can choose various filters. Selecting a filter in the leftmost box (e.g. clicking on an album name after you've clicked the Album tab in that box) filters the contents of the left two listboxes. In this way you can drill down by choosing up to three filters. E.g. "Rock" in the left list from the Genre tab, then 1988 in the middle list from the Date tab, then Money for Nothing in the right list from the Album tab. This is kind of a clumsy approach to filtering, but in practice you don't need more than three filter selections to narrow even a large library significantly. The filters are a clumsy but reasonable approach. The difficulties start with the bells and whistles. You can right click on any of the three listboxes for a context menu.
- There's no other way to access the features in the context menu except by right clicking on the list. But right clicking on the list selects the list item where you clicked -- a minor annoyance.
- One of the menu options is "Show Buttons", which makes a toolbar appear below the listbox. Seriously? Why not just show the toolbar all the time?
- A minor point: one of the buttons is a broom icon for "Clear Selection". When you click this, the selected item in the listbox above is deactivated. But the item is still selected. The selection should instead be removed.
- The buttons that are displayed depend on which tab is selected. This is nonstandard: normally you disable a button instead of hiding it if the functionality is unavailable for the current mode.
- There's an "Intersection Mode" toggle button. I haven't been able to figure out what this does; nothing meaningful as far as I can tell.
- There's an "Options" button that displays a menu of display options.
- There are a confusing array of display options, all of which depend on the currently selected filter tab. When looking at the Date tab, for example, you can choose between a standard list or "cloud" mode. (I'll admit that having cloud mode available is kind of nice -- I can see 30+ years all at once instead of having to scroll.) If I switch to the Album tab, there are many more choices: six different sort modes; optionally displaying year, song count and running length of each album; five choices for showing album pictures and size to show; and the choice of cloud or "mosaic" mode. Mosaic mode is like "Icon View" in Nautilus -- albums are listed left-to-right with art shown where present and text where no image is available.
- I won't get into all of the behavior here, but the bottom line is: too many modes, which are not orthogonal, and your options disappear depending on mode -- without having a clear path to getting back to the option you were using.
Just below the listboxes (and optionally the listbox-specific toolbars) is another toolbar with a filtering control, a few buttons, and some menus. The menus don't belong here: they belong in the app's main menu bar.
The "Saved List" feature is completely unusable. There's a menu option "Save current list as" that can only be found on the Saved tab in the filter listboxes -- but you have to be careful to click on the "Saved Lists" item in that list, because it's the only place you can right click that won't alter the filter contents displayed in the main window. I don't know if the feature is buggy or what, but clicking on a saved list doesn't consistently display the contents of that list.
Overall, I'd put it down to a bad case of featuritis. I don't think there's a widget that hasn't been used in the UI, things are in nonstandard places, features are hidden in context menus, and there are way too many unexpected behaviors.

If I were going to implement a similar music player with advanced, highly flexible filtering; playlist construction; an ad-hoc queue; and flexible playback sequencing (e.g. in-order with various sorts, or weighted random according to various criteria) I would:

Make three explicit playback modes: queue, filter, or playlist. You can edit any object regardless of which is playing, but, for example, changes made to the queue don't affect playback in playlist mode. This makes it easier for the user to figure out what he's currently working with -- and would actually enable new features like "save current queue to playlist".
Similarly make three explicit editor/browser modes. You're editing a specific (i.e. named) filter or playlist, or you're modifying the queue.
The player and the browser/editor would be distinct. Changes in the browser don't implicitly change what's happening in the player (except for the queue -- which you expect to make fluid changes to the player). If you're editing the actively playing playlist, you have to explicitly save your changes to the playlist for them to take effect. There are no editing controls in the player (e.g. moving songs up and down in the list).
Retain some of the context menus where they apply to the selected object (e.g. "Song Properties" when you click on a song in a list), but I would move many options into the main menu.
Use a more familiar UI for building filters. Something like what you see in the "Rules" window for email clients, for example -- AND/OR selections along with different filter rules. This would let the user build filters just as complex as gmusicbrowser allows but easier to navigate (in fact, gmusicbrowser has a window with this sort of interface).
Retain (and expand) the status information: song index, total song count, total play time, remaining play time, etc. Probably in a status bar, but possibly in the header of the player.
I didn't mention it above, but I'd retain the awesome bulk modify feature with some tweaks for usability. (Feature is cool, usability in gmusicbrowser is fair to poor.) This feature allows you, for example, to rename a whole batch of files according to a template, e.g. "<ALBUM> - <TRACK#> - <TITLE>.mp3". It also allows you to make bulk changes to the ID3 tags on a bunch of files all at once.
Integrate with mmkeys.
Minimize to the tray, and possibly have a shrunken player window that just shows a standard set of stop/play/pause/forward/backward icons, title, etc -- instead of needing a big library browser and/or playlist window open all the time.

Maintaining "Privacy" Under the New Google Privacy Policy

2012-02-02T21:59:00-05:00

I put "privacy" in quotes, because under the scheme below you're still giving up a lot of data, but you can (I think) still maintain "silos" of data so that, for example, your search data isn't correlated with your YouTube or GMail data.

See What Actually Changed in Google’s Privacy Policy from the EFF for a good description of the substantive changes in Google's new privacy policy.

Here's what I'm doing to try to preserve some of my data in silos. This is based on my understanding of the way the new privacy policy works, which could be faulty. Remember that you're still giving up your data by using these services.

In Firefox, I've installed Duck Duck Go as a search engine, and I use it as the default. I've been using this for months now, and it doesn't suck. The results are almost on par with the results I used to get through Google, and they have a surprisingly "private" privacy policy.
In Firefox, I've installed Ghostery. This blocks all kinds of trackers, including Google Analytics.
I've switched from using the GMail web UI to a desktop client. Google still sees all my email, but I don't have to remain logged in to that account in the browser. (See #4.) This effectively makes the mail account a silo; none of the data is used elsewhere. I occasionally have the fantasy that I'm going to move my domain off Google Apps and go back to self-hosting, but that's unlikely to happen in the near future.
When I'm browsing, I'm logged into a second Google account that is disconnected from my primary mail account. (See #3.) This means that, for example, videos that I watch on YouTube aren't correlated to the content of my mail.
In Firefox, I've installed Adblock Plus. This cuts down on tons of ads; which might avoid leaking your data to advertisers.
I haven't done it yet, but I plan to find a desktop RSS reader that doesn't suck and delete everything from Google Reader.
I've stopped checking in at Google News. Instead I pick one of CNN, News360, BBC, or other news outlet.
I'm not sure what it would be, but if I'm browsing stuff that I don't want tracked, I'll fire up a Chrome Incognito session and proxy everything through Tor. (The only reason for using Chrome is that I typically don't want to interrupt a Firefox session to drop into Private Browsing mode there.)

It's worth noting that I don't use G+, so I'm not doing anything specific to maintain a silo there.

As noted above, most of this doesn't vastly enhance your privacy. You're still handing out a lot of data every time you browse the web; it's a fact of life. And it is, after all, very convenient to have access to your search history -- I often find myself trying to figure out what that thing was that I found during a search a couple of weeks ago but can't remember the terms I used. But I'm hoping that it cuts down on the amount of data correlation between my various online activities.

Turning Photos to a DVD Slideshow

2012-01-24T10:36:00-05:00

The task: take a bunch of photos from my digital camera and burn them to a DVD that would play the photos as a slideshow in any DVD player. (Actually, the task was to figure out how to do this in the simplest possible way, and make a set of instructions for an unsophisticated user to follow.)

The result: several hours of fighting with a dozen different applications before I actually had photos displayed on my TV. And this post.

I don't understand why what seems like a fairly simple task is so complicated.

This is what I found to be the easiest recipe on a stock Ubuntu 11.10 installation. Fair warning: this is quick & dirty, you don't get special effects, you don't get music, etc.

Required software (load through the software center, or via apt-get): imagination, devede, brasero.
Open Imagination. Click the import photos icon (it is a terrible choice of icon -- there's no clue that you should want to click on the black square to import photos) and choose your photos.
Export video. Choose DVD for output; to keep it organized, drop the video file in the Videos directory in your home directory. This can take a while if you have a slow computer and/or a lot of photos.
Open DeVeDe. Configure the title. Import the video you just generated.
Generate the DVD ISO image; again save the image to your Videos folder.
Insert a DVD in the drive. Open your Videos folder. Right click on the ISO image and choose Write to Disc.

Some notes, in no particular order:

If you want to tweak the slideshow, imagination will let you play around with transitions, slide duration, etc.
If you want to import video clips, use Openshot instead of Imagination. It's harder to use (and overkill) compared to Imagination if all you want to do is produce a static photo slideshow. But if you want to do any video editing, it's a good combination of power and ease of use.
If you want music, Openshot might also be your best bet. I didn't look at adding music via Imagination or DeVeDe.
Don't try to use the Brasero application directly. I experienced crashes/lockups while using it, but it worked well when all I did was right click the ISO and choose Write to Disc. (It's also an ugly, hard to use piece of junk.)

Things I tried and rejected:

K3b: After Brasero failed miserably twice, I tried K3b. Rejected it in favor of not pulling in a bunch of KDE infrastructure that I wasn't using anyway, and also in favor of the simplicity of just directly burning the file without having to explicitly open a separate app.
dvd-slideshow: This command line app does appeal to my inner geek, but I can't recommend it for users who want a GUI. (Even the inner geek appeal isn't enough to make me want to use it when the combination outlined above works well.)
Mistelix: As good as Imagination, and I would have selected it if I had quickly found a way to rotate a photo from within Mistelix. Photos that showed up auto-rotated in other apps showed up sideways in Mistelix and I didn't see an obvious rotate button. Imagination did the right thing.
Bombono: Junk. Don't bother with it. I opened it, saw that the tabs were overlapping the menus, couldn't figure out how to import any photos, and purged it from my system.
Videoporama: Development is dead. It wasn't immediately obvious that it was going to be easy to use for the simple task of creating a DVD from a set of photos.

The software stack above would be 10x better if Imagination implemented a feature to export a DVD ISO. It would be 100x better if it had a "Burn DVD" button that spit out a shiny disc that I could drop into a DVD player.

(To be fair, the whole video situation is 1000x easier than it was 5 or 6 years ago when I tried to do something similar, realized how much work it was going to be, and walked away in frustration. The tools have come a long way.)

Tightening UFW Firewall Rules to Limit SSH Access

2011-12-08T18:54:00-05:00

The auth.log on one of my servers (really, on all of the servers I have access to) is full of stuff like this:

Dec  8 03:19:33 localhost sshd[4718]: User root from 10.1.2.3 not allowed [...]
Dec  8 03:19:35 localhost sshd[4721]: Invalid user db2inst1 from 10.1.2.3
Dec  8 03:19:38 localhost sshd[4723]: User root from 10.1.2.3 not allowed [...]

fail2ban is configured to (temporarily) block these after a certain number of attempts, but they keep coming back. One particular IP address was hitting ssh constantly (except for the ban periods) for several days, so I added a rule to drop everything from that address -- but this strategy isn't scalable.

The simple, obvious solution is to only allow access from known hosts. It would be very rare that I need to access this server from anywhere except a small number of addresses. I started with a ufw (Ubuntu's "uncomplicated firewall") ruleset like this:

% sudo ufw status
Status: activeTo                         Action      From
--                         ------      ----
Anywhere                   DENY        10.1.2.3
Anywhere                   DENY        172.16.99.88
22                         ALLOW       Anywhere

To avoid getting locked out, I added (temporarily) a crontab entry for root:

*/15 * * * * /usr/sbin/ufw allow ssh

This will allow access to ssh from any un-banned IP address (the current policy) every 15 minutes. Then I inserted a rule that allows access to ssh from my home ISPs netblock. (You can figure out the netblock by doing a whois lookup on your external IP address; you may need to add multiple netblocks if your ISP has several allocated. Be careful: it's no fun to get locked out!) This action is safe because I have not yet removed global ssh access. Note that I'm using "insert 3" to add this rule at a specific position in the list.

sudo ufw insert 3 allow proto tcp from 10.9.8.0/18 to any port 22

And I added a rule to permit access from another server with a fixed IP I have access to (for the rare case where I need to access this server when I'm not at home):

sudo ufw insert 3 allow proto tcp from 172.17.101.102 to any port 22

Now my ruleset looks like:

Status: activeTo                         Action      From
--                         ------      ----
Anywhere                   DENY        10.1.2.3
Anywhere                   DENY        172.16.99.88
22/tcp                     ALLOW       172.17.101.102
22/tcp                     ALLOW       10.9.8.0/18
22                         ALLOW       Anywhere

So far, so good, but ssh access is still permitted from anywhere -- because of that last rule. This is the dangerous part... you could get locked out if you haven't set the rules correctly. (You set up that crontab entry, right?)

sudo ufw delete allow ssh

Wait for the prompt... hooray, I'm still connected! After checking that I can access ssh from home and the other server, I know it's safe to remove the crontab job. (If the cron job has already fired, you'll need to rerun the ufw delete allow ssh command.)

At this point I can delete the first two rules that ban specific IPs, since they're outside my netblock and won't be allowed anyway.

Now I can enjoy quieter logs without all those access attempts from China and Croatia!

Bug Bounty Programs

2011-11-17T19:48:00-05:00

There are several companies offering "bug bounties" -- payments to people to find security bugs and report them responsibly.

I will update this page with links to the various bug bounty programs as I become aware of them. If you know of a program that is not linked here (or if something below becomes stale) please leave a comment below or send me a note. The notes below are a summary -- read the fine print in each program.

In no particular order:

Mozilla offers $500 to $3000 (plus a t-shirt!) for serious security bugs in Firefox, Thunderbird, or certain Mozilla web apps.
Facebook offers a "typical bounty" of $500 for responsible disclosure of security bugs (XSS, CSRF, injection, etc) on Facebook, not including third party applications.
Tarsnap, an open-source encrypted online backup service, pays a sliding scale of $1 for spelling errors in code comments to $1000 for "a bug which allows someone intercepting Tarsnap traffic to decrypt Tarsnap users' data".
Google's vulnerability reward program pays a base of $500, up to $3133.7, for "any serious bug which directly affects the confidentiality or integrity of user data" (e.g. XSS, CSRF, authentication, etc) on Google properties (youtube, orkut, *.google.com, etc).
Chromium has a vulnerability rewards program (sponsored by Google, of course) that pays a base of $500, up to $3133.7, for "High and Critical impact bugs".
Barracuda Networks has a Security Bug Bounty Program that pays $500 to $3133.70 for bugs "that compromise confidentiality, availability, integrity or authentication" in certain Barracuda products.
Piwik pays $500 for critical security bugs, and $200 for non-critical bugs.
Artifex has a bug bounty program for ghostscript that does not appear to pay for reports, but does pay $1000 for patches that fix bugs in their tracker flagged P1 or P2 and "bountiable". They pay $500 for lower priority bug fixes.
One of the oldest programs I'm aware of is Donald Knuth's offer of $2.56 for reporting errors in his publications. (Note that he's no longer sending out checks due to problems with check fraud. Here, the intrinsic reward is worth more than the money anyway.)
Hex-Rays pays $3000 for security bugs in the IDA or decompiler.
White Fir Design has a Wordpress bug bounty that pays a sliding scale, from $50 for privilege escalation to $500 for remote execution of arbitrary PHP or malicious file injection in Wordpress, and $50-$250 for bugs in plugins with one million downloads.
White Fir Design has a Drupal bug bounty that pays the same sliding scale as their Wordpress program ($50-500 for Drupal core code, $50-250 for contributed modules).
TippingPoint's Zero Day Initiative pays reports of vulnerabilities in third party products. Whether they pay depends on how widely deployed the product is, and the severity of the vulnerability. Their program is designed to reward repeat vulnerability reporters. (E.g. if you report 10 bugs worth $1000 each, you get a $1000 bonus and 10% bonus on all subsequent reports for a year.)
Harry Tenant and Associates pays $5 for bugs reported in School Site Manager.
Ricebridge offers $15 per bug reported in any of their products.
Microsoft is running a BlueHat Prize Contest from 2011-08-03 to 2012-04-01. It's a contest, not a bounty, and only two entrants will receive cash: $200k and $50k for first and second place, respectively.

Using SSH for IPv6-enabled HTTP Proxying

2011-11-11T11:50:00-05:00

My ISP has apparently made no progress whatsoever with IPv6, but I've got an IPv6 enabled VPS.

SSH makes it trivial to use that VPS as a SOCKS5 proxy. Just do:

ssh -D 8080 myvps.example.com

Then set your browser's SOCKS proxy to localhost:8080. In Firefox on Linux, this is Edit > Preferences > Advanced > Network tab > (Connection) Settings > Manual Proxy Configuration. Leave all fields blank except for SOCKS Host and Port -- localhost and 8080, respectively. Choose SOCKS5. Then browse to about:config and change change network.proxy.socks_remote_dns to true. This tells Firefox to ask the proxy to resolve names instead of trying to resolve them using your ISP. Chrome worked for me without hassle.

Go to test-ipv6.com to test that it works. If your results from test-ipv6.com indicate that IPv6 name lookups are failing, make sure you've got that about:config setting mentioned above changed.

Fixing Certificate Errors with Cisco AnyConnect

2011-11-06T10:50:00-05:00

"AnyConnect cannot confirm it is connected to your secure gateway. The local network may not be trustworthy. Please try another network."

There may be several reasons for this error, which you'll find on other pages that hit for a search on this string.

The reason that I encountered seems to be unique. What I found by digging into a wireshark capture is that AnyConnect sends a TLS alert to the server, disconnecting the session. The alert message says "Unknown CA".

It could have something to do with installing the firefox plugin "Certificate Patrol" recently. AnyConnect apparently uses firefox's certificate store. Perhaps Certificate Patrol does something to the store that makes it so that AnyConnect can no longer use it?

In case it matters, I'm on Ubuntu 10.04.

Whatever the cause, you can fix it by doing the following (Edit: see also a simpler method in a comment by Nathan below):

Figure out the CA that signs your VPN server's certificate. (Hat tip to Didier Stevens for the easy way to do this.)
1. Fill in your server, I'm going to use www.google.com.
2. openssl s_client -connect www.google.com:443 >! /tmp/google
3. Hit ctrl-C.
4. You saw "Thawte Consulting (Pty) Ltd.".
Now search for "thawte root certificate". (Obviously this will vary depending on who signs your server's certificate.)
Go to the page where they list certificates. Download them all. (I tried downloading the one that looked like it matched the signature on my server's certificate, but I think you need everything down to the root in order to fully verify it. It's easier to just download everything than try to figure out exactly which ones are needed.)
Convert to PEM format. (Hat tip to Mozekoze for this recipe.)
1. openssl x509 -in input.crt -out input.der -outform DER ## (if the certs are in .crt format)
2. openssl x509 -in input.der -inform DER -out output.pem -outform PEM
Copy all the PEM files to /opt/.cisco/certificates/ca. (Or to ~/.cisco/certificates/ca.)

Alternatively, you could copy the certs out of the keystore on your machine, convert to PEM, and then copy the PEMs into the directory mentioned above. (On ubuntu, you can see the certificates in the package ca-certificates. You could copy just the thawte certs by doing something like cp $(dpkg -L ca-certificates | grep -i thawte) /tmp/certificate-conversion/.)

Three cheers for wireshark, strace, openssl, and google for help figuring out what was going on. Cisco could make things a little bit easier to figure out -- a decent diagnostic message would have been great -- just tell the user "Unknown CA", it's already buried in the wireshark trace.

Preserve Your Sanity When Dealing with Unicode in Python

2011-10-05T11:09:00-04:00

I have not yet played with python 3, which sounds like it makes working with Unicode easier.

In python 2.x, working with Unicode can be annoying, but when you remember this rule of thumb, it's much easier and allows you to keep your sanity:

Always use unicode strings internally. Decode whatever you read/receive. Encode whatever you write/send.

I've recently done some work on a project to generate ReStructuredText (as an intermediate form on the way to generating HTML). The input data has Unicode sprinkled throughout (in UTF-8). I kept getting UnicodeDecodeError: 'ascii' codec can't decode byte exceptions in various places, until I applied that rule everywhere:

When reading, decode from UTF-8.
Use Unicode strings u'E.g. this formatted %s string' % (decoded_string1, decoded_string2) internally -- everywhere.
When writing, encode to UTF-8.

Hat tip to nosklo on StackOverflow for mentioning this simple rule.

My First Android Adventure

2011-09-26T21:43:00-04:00

Here are my notes about getting the Android SDK up and running using only the command line tools that come in the SDK.

The following worked for me on Ubuntu 10.04 LTS as of September 2011. I find Eclipse to be an excruciating form of torture, so I'm not following the Eclipse-based instructions. Everything below can be done from vi or emacs and the command line.

Installation

Download the SDK.
Unpack the SDK.
Run android-sdk-linux_x86/tools/android to bring up the AVD Manager. (AVD is "Android Virtual Device" -- it will run in the emulator.)
Select "Available Packages", and expand "Android Repository". It will fetch a list of packages.
Select "Android SDK Platform-Tools, Version 7". (You need this.)
Select "SDK Platform Android 2.2, API 8, revision 3". (You need at least one platform. My development phone has Android 2.2, so that's what I'm initially targeting. You can add more platforms later.)
I also selected Documentation and Samples, for convenience.
Install the packages you selected. Restart ADB when prompted.
Then create a new virtual device. In the AVD Manager, click "Virtual Devices", then "New...". Give your device a name, choose the platform, click Create.
Install the ubuntu package "ant1.8". Don't install "ant" -- this is version 1.7, which won't work with the android SDK.

Create a New Project

I unpacked the tools under ~/projects/android.

From that directory, I can give this command:

./android-sdk-linux_x86/tools/android create project \
  -p ./hello_android -t android-8 -a HelloAndroid \
  -k "com.blakitasoftware.hello_android"

This creates ~/projects/android/hello_android/ containing the new project.

Build the Project

Change into the hello_android directory. Run ant debug.

(Read this for info on package signing and building in debug mode.)

Run the Application

Since we built in debug mode, we don't need to worry about signing -- we can run it in the emulator right away.

First start the virtual device: run android with no arguments, find your device, click "Start...", "Launch".

Install the application by running (assuming you're still in the hello_android directory):

../android-sdk-linux_x86/platform-tools/adb \
  install ./bin/HelloAndroid-debug.apk

Then go to the emulator, click the Launcher. (This is the grid of little gray squares in the bottom of the screen. Yeah, it took me a few seconds to figure this out -- I've never even used an Android phone, this really is an adventure...) Find your HelloAndroid app, click it, and you'll see the greeting ("Hello World, HelloAndroid").

Make a Change

To change that greeting, edit res/layout/main.xml.

Assuming you only have one emulator running, the build and install sequence given above can be abbreviated to ant install. This reinstalls your app with the new greeting, find your app in the launcher again and rerun it to see the change.

Change directories faster with zsh

2011-07-25T10:23:00-04:00

Something I didn't know until recently:

zsh does not require cd to change directories.

Using the directory as a command implies "cd". For example, instead of doing:

me@server:/home/me $ cd /tmp
me@server:/tmp $

You can just do:

me@server:/home/me $ /tmp
me@server:/tmp $

That's three whole keystrokes (nearly half the command shown above), all day long.

Bringing up a New Cloud Server

2011-07-21T10:08:00-04:00

This is a brain dump of my task list when bringing up a new server. (For my purposes, a linode running some version of Ubuntu.)

Create a user account.
1. Add user to wheel.
2. Set up sudo for wheel.
3. Set up ssh authorized-ids.
4. Change shell to zsh.
5. Copy dot-files.
SSH config file tweaks.
1. Disable SSH root logins.
2. Disable SSH password logins (key only).
3. Disable access for non-ssh-group users.
4. Disable DNS lookups (UseDNS no).
5. Other?
Move sshd listening port to nonstandard high port. Test that logins still work! (I'm not convinced this buys much -- see firewall & fail2ban at number 8 below.)
Change root password.
Set the hostname.
Set up DNS entry(-ies) (forward & reverse).
Download OS updates. (Configure repos / mirrors as needed first.)
Activate firewall (ufw).
- Add rule for SSH.
- Install fail2ban and configure to work with ufw.
Set the time zone (dpkg-reconfigure tzdata).
Set up an offsite backup. (rsnapshot or other)
Set locale.
Reboot -- make sure ssh logins work after rebooting, etc. (Easier to fix problems now than when you need to log in at some later time.)
Install packages for whatever purpose the machine is going to be used. (Apache, git, etc.)
Portscan the server (externally is preferred) to make sure there are no leaks.

I'll come back to this list occasionally to revise it. Please leave a comment if you think something important is missing...

Use Linux to build win32 installers for Python apps

2011-07-20T10:00:00-04:00

A python-based project I'm working on has a win32 user that I need to support. Until yesterday I've been moving to a win32 laptop in order to run python setup.py bdist_wininst so I can produce a self-installing executable. Then I discovered how trivial it is to use wine to do the job:

Install wine. (sudo aptitude install wine)
Install python into the wine environment. (Download an msi from python.org and run msiexec /i python-x.x.x.msi.)
Install whatever prerequisite packages you need (e.g. wxPython) using wine or msiexec.
When you've got everything ready to build, just do wine c:/Python27/python.exe setup.py bdist_wininst and look in ./dist/ for your exe!

Making perl unit tests easier to run in emacs

2011-07-07T19:57:00-04:00

It's a hassle to have to switch from emacs to the shell, run unit tests, pick out the failure, switch back to emacs, navigate to the line, forget where you were supposed to go, switch back, etc etc.

The whole process is much easier with compilation-perl.el. This handy bit of code sets up patterns for compilation-mode so that test failures (among other things) are recognized. Just add compilation-perl.el to your emacs lisp directory and follow the directions at the top of the file for installation instructions.

So my test process is now:

Hit F4 (which I have mapped to compile).
If necessary, change the command to either make test or to run a specific test I'm working on (e.g. perl t/MyModule.t).
If there's a failure, press C-F4 (which I have mapped to next-error) and it brings me right to the failure.
Repeat from step 1 (but the second time around I don't need to do step 2 since the command is already set properly).

Make apt-get use an alternate sources.list

2011-01-06T11:18:00-05:00

Since my [former] company's product is based on ubuntu, we use apt to distribute upgrades. We don't normally want customers to pull upgrades from the upstream distribution, so we've pruned back the main /etc/apt/sources.list to include just our server. We rename the original default sources.list so that it doesn't get picked up.

But sometimes I want to be able to pull from upstream, so I want to reactivate the original sources.list. Up to now, I've been having to do an awkward dance that involves renaming our file, copying the old default to the active, and then doing the reverse when I'm done.

(As an aside, yum-based systems like Centos or Fedora make this easy. You can say enable=0 in the list file, and then use --enablerepo=myrepo on the command line. There doesn't seem to be a way to do this with apt, but the command below is almost as good.)

To avoid this renaming dance, here's a better solution.

First, create a directory to put the original sources.list: /etc/apt/sources.list.ubuntu.

Then copy the original default sources.list into that directory. I called it /etc/apt/sources.list.ubuntu/jaunty.list.

When you want to pull from upstream, run:

apt-get -o Dir::Etc::SourceParts=/etc/apt/sources.list.ubuntu install some-new-package

Then you'll want to run apt-get update to reset your repository list back to the trimmed-down set.

An Interesting pid File Race

2010-05-26T11:08:00-04:00

ISC's dhcpd uses this code to check for an already-running daemon:

/* Read previous pid file. */
if ((i = open (path_dhcpd_pid, O_RDONLY)) >= 0) {
    status = read (i, pbuf, (sizeof pbuf) - 1);
    close (i);
    if (status > 0) {
        pbuf [status] = 0;
        pid = atoi (pbuf);        /* If the previous server process is not still running,
           write a new pid file immediately. */
        if (pid && (pid == getpid() || kill (pid, 0) < 0)) {
            unlink (path_dhcpd_pid);
            if ((i = open (path_dhcpd_pid,
                           O_WRONLY | O_CREAT, 0644)) >= 0) {
                sprintf (pbuf, "%d\n", (int)getpid ());
                write (i, pbuf, strlen (pbuf));
                close (i);
                pidfilewritten = 1;
            }
        } else
            log_fatal ("There's already a DHCP server running.");
    }
}

The problem with this strategy is that, if the box dies, there's a stale pid file left in /var/run/dhcpd.pid. This wouldn't be so bad -- the code above checks [using kill(pid, 0)] to see if there's a process running with that pid. But when the box is restarting, there will be a bunch of processes all starting in similar sequence each time. So on one boot, you might see dhcpd with a pid of 1001 and ntpd with a pid of 1002. If the box dies violently (e.g. power cut), the dhcpd pid file will contain 1001. On the second boot, assume ntpd starts first and gets a pid of 1001 and dhcpd is 1002. Now, the kill(pid, 0) will succeed, making it appear that dhcpd is already running, and dhcpd will exit.

How to fix this?

Explicitly put the pid file under /tmp. Getting this right is fussy -- make sure you avoid the race conditions associated with creating temp files. Use dhcpd's "-pf" flag to tell it where to use the pid file. This avoids spurious "already running" messages, because dhcpd will never read a pid from an existing pid file. [You could also just remove the /var/run/dhcpd.pid file, but I'd rather explicitly provide the path in my startup script in case some dim bulb decides to change the compiled-in default.]
Be careful in your restart code to kill any existing dhcpd (assuming you really want a new dhcpd), or avoid trying to start a new one (assuming you want to use an already running dhcpd). pgrep(1) and pkill(1) will be useful here.

In researching this, I saw this bit of wisdom from Henning Brauer: "pid files are useless.".

I heartily agree...

Using Lenny to Build Jaunty Packages

2010-05-13T10:43:00-04:00

The process for building debian packages is actually very well documented, though many of the tutorials you will find are aimed at people who are packaging third-party software instead of their own.

Also, I'm running on Debian's Lenny, and I need to produce packages that install cleanly on Ubuntu's Jaunty Jackalope. It sounds worse than it is.

Take this with a grain of salt --- it works for me, but I'm certainly not an expert in this area.

Steps:

Debianize each package. (Here's an excellent quick and dirty guide to getting a package created.)
Build the packages: (cd MYPROJECT; fakeroot ./debian/rules binary). Don't distribute these packages, they are probably wrong! But this is the quickest way to get everything mostly-working.
Sanity check the packages: dpkg-deb -c MYPROJECT.deb; dpkg-deb -I MYPROJECT.deb; dpkg -i MYPROJECT.deb
Check in your MYPROJECT/debian/ directory to source control (e.g. git, svn, etc).
This is the tricky step, and one that I didn't see explicitly documented.You need a version of debootstrap that knows about ubuntu distributions. (Ubuntu debootstrap knows about Debian, but not the other way around.)
1. Download the jaunty (or whatever ubuntu version) debootstrap source package. Grab the .dsc and .tar.gz from that page.
2. Run dpkg-source -x debootstrap_1.0.20~jaunty1.dsc -- this extracts the source.
3. (cd debootstrap-1.0.20~jaunty1; debuild -uc -us) -- this will build a .deb you can install on your lenny machine. Install package "devscripts" if you don't have debuild.
4. sudo dpkg -i debootstrap_1.0.20~jaunty1_all.deb
Install pbuilder. This will help you build the packages in a clean chroot -- i.e. without all the pollution you have on your machine that could be making the packages work even though they will break on your users' machines.
Put the code found here in your ~/.pbuilderrc.
Apply this sudoers change (optional but convenient).
sudo DIST=jaunty pbuilder create ... then wait. (If you have lame internet service (Hughes) and a daily bandwidth cap, you may want to schedule this for the free-for-all period: echo sudo DIST=jaunty pbuilder create | at 2:30am tomorrow)

If you see anything I'm missing or would like to add, please drop a comment below. Thanks!

zsh history expansion

2010-02-15T08:59:00-05:00

Exploring zsh features made me want to figure out some of the history-editing wizardry. (Bash has similar history tricks, I just never bothered to dive too deeply into them.)

If you want to experiment with history expansion a bit, you can echo the result instead of executing it:

hostname:~/dir% ls /some/long/path/to/file_0.1-2_i386.changes
hostname:~/dir% echo !?ls?:s/-2/-3/
echo ls /some/long/path/to/file_0.1-3_i386.changes

In this case, what I wanted to do is repeat a long command that referenced a file with a version number in it --- but I wanted to use a different version number (-3 instead of the previously given version -2).

History expansion uses "!" ("bang", or exclamation point). If you want to select the previous command to expand based on a string match, "!?str" will find the previous command that contains "str". If you want to substitute some part of that previous command, put another question mark on the end, add a colon, and use the the "s" modifier. In the example above, "!?ls?" means expand the previous command containing "ls". The ":s/-2/-3/" means modify that expanded command to replace the occurrence of "-2" with "-3".

(Of course, to actually get all of that figured out and working, I had to iterate the sequence of commands above about a dozen times.)

Leave a comment with your favorite zsh history trick. Thanks!

Data vs Code

2010-02-05T11:08:00-05:00

I'll take an array over a giant switch-case statement any day.

The array definition will be more compact and easier to see all at once.
Defining actions in an array enforces uniformity.
You can put checks in the code to automatically verify that the array definition is complete. (I.e. verify it contains a definition for every item it should have.) Yes, some tools can do this for certain types of switch statements. Using an array-based check is more portable and more foolproof.

If you have a switch statement of mostly cut and paste cases, you can probably convert it to an array very easily, and then rewrite the switch statement to look up the value in the array and do whatever thing is supposed to be done, either via function pointers or by using an associated value from the array.

I did this on a horrible switch statement once. Dozens of cases, 80% were nearly identical but the few that weren't were awful to untangle. Once I pulled the case bodies into separate functions, put function pointers into a table, and replaced the switch body with a lookup loop it was much cleaner. The code for the odd cases was eventually pushed out (it was a symptom of bad design). The whole exercise enabled another round of changes that allowed the functions for the case bodies to be collapsed back into the table -- we ended up removing an entire unnecessary layer of indirection and made the design much easier to grok.

Set Your zsh Prompt

2010-02-01T11:05:00-05:00

Since the beginning of time, all the cool kids have had really cool shell prompts. It's a great place to display helpful information, and zsh has features that let you have a flexible, informative, unobtrusive prompt.

Set your prompt by setting $PROMPT. If you do PROMPT='foo ', the shell will give you a foo prompt for every command. Not terribly useful but you get the point.

There are a bunch of codes you can use in the value of PROMPT to get useful output. For example, %m gives the name of the machine you're running on, and %~ gives the name of the current working directory. For a list of all the codes, check the "zshmisc" man page under the section "SIMPLE PROMPT ESCAPES". Note that there are codes for boldface, underline, and colors here too.

The drawback to having all kinds of information in your prompt is that you limit the length of commands that you can enter without scrolling. Scrolling stinks because the command is harder to read. Enter the "right prompt". Set RPS1 to contain the lengthy part of your prompt, and it will be displayed on the right margin of your terminal. Then when commands get long and encroach on the right prompt, it will conveniently disappear. Read Quentin's comment on this zsh post.

Here's a gotcha with RPS1 and fancy prompt formatting: gnome-terminal does not behave well when these are set. I'm trying out alternatives to see which will be better.

Lastly, a trick I picked up from Justin. Zsh provides hooks that you can use to do things before and after a command runs. Just define the functions preexec and precmd. This example sets the title of the xterm while a command is running:

function title() {
    # escape '%' chars in $1, make nonprintables visible
    local a=${(V)1//\%/\%\%}

    # Truncate command, and join lines.
    a=$(print -Pn "%40>...>$a" | tr -d "\n")
    case $TERM in
        screen*)
            print -Pn "\e]2;$a @ $2\a" # plain xterm title
            print -Pn "\ek$a\e\\"      # screen title (in ^A")
            print -Pn "\e_$2   \e\\"   # screen location
            ;;
        xterm*)
            print -Pn "\e]2;$a @ $2\a" # plain xterm title
            ;;
    esac
}

# precmd is called just before the prompt is printed
function precmd() {
    title "zsh" "%m:%55<...<%~"
}

# preexec is called just before any command line is executed
function preexec() {
    title "$1" "%m:%35<...<%~"
}

Update: Set variable a to local in function title above as suggested in the comment below.

Insist on Automatic Tests

2010-01-29T14:08:00-05:00

At some point your team is going to be gone. Not all at once (well, maybe, but in that case you won't care), but over time turnover will completely replace your team.

If you are the manager, and you outlast the team, you're going to pay for low quality code when you try to bring in new people and they end up breaking everything becuase there's no tests to check their bug fixes and/or enhancements.

If you are a surviving member of the team, you're going to pay with huge headaches because you've got to fix code where you have no idea what might be broken as a result.

Change is scary when you don't have automatic tests. Just suck it up and write the tests!

Make the zsh zle handle "words" correctly

2010-01-27T10:40:00-05:00

Snippet from my .zshrc:

# This controls what the line editor considers a word. By default it
# includes '/', which makes it so that when I M-del (attempting to erase
# a directory in a path), I erase the whole path. Annoying.
# WORDCHARS='*?_-.[]~=/&;!#$%^(){}<>' # (default)
WORDCHARS='*?_-.[]~=&;!#$%^(){}<>'`

After living with this for a while, I realize that I should probably remove the underscore too, since that's what I was used to in bash.

Moving to zsh

2010-01-25T10:20:00-05:00

To get started:

sudo aptitude install zsh
chsh /bin/zsh

That's pretty simple.

Of course, you're not running zsh yet... either logout and log back in or just run zsh at the prompt. You'll get a series of prompts to configure a .zshrc. It only takes a few minutes, so run through the options and save the file.

Next up: setting a custom prompt.

9 "Must-Have" Tools for Software Teams

2010-01-18T11:06:00-05:00

The items below are useful systems based on my experience working with a bunch of different software teams at a handful of companies over the past decade-plus. I haven't bothered to list things like compilers, interpreters, libraries, etc. If you don't have those, you aren't making software...

Source control. This almost belongs in the "if you don't have it, you aren't making software" category. There are so many free and simple to use tools that it is ridiculous to not use source control. Pick one of: subversion, git, or any of the dozens of other free tools and use it. I guess I'd say "if you don't have it, you are making a mess".
Bug/issue tracker. Trac is a decent tool for keeping tabs on development tasks and bugs. Bugzilla is ok too. If you have a budget, there is an apparently infinite set of commercial systems available of varying quality. Almost anything is better than a list of scribbles on your whiteboard.
Backup server. You can do software without it, but the odds are against you lasting long.
Build system. Automated nightly/continuous builds are awesome. Whether you DIY, install something free like hudson, or purchase a commercial system, this is a key piece of making sure that the software maintains at least a minimum level of quality (i.e. it is always buildable). After having worked on a couple of teams that didn't have such a system, and the build was always broken, I would never work without it again.
IM / Jabber. Especially with a conference server where the rooms are archived and available via the web. Searchable is nice, but making them available where they can be grepped isn't bad either. I didn't become convinced of the value here until last summer.It sounds like an extra distraction and sometimes it can be, but you can turn it off when you're heads-down coding. And it has the potential to reduce the number of interruptions from people walking into your office with questions that could have been answered in an IM. It's lighter weight than email and rooms are broadcast by nature so you don't have to spam a huge list of people to find the answer you're looking for.
Email lists. Must have web-browsable archives; the list without this has much less value. Search would be a nice bonus, though I haven't seen it done well. This lets teams communicate a bit more formally than IM and provides a record of certain discussions and decisions.
FTP server. Bonus points for a TFTP server. Maybe it's because of the nature of embedded software, but every team I've been on has always had to fling around large files and images. Email is horribly inefficient for this. Also, most of the products I've worked on needed an FTP (or TFTP) server to boot from. You're likely to install something on your development machine, which is fine, but it's convenient to have a central team/company internal server so you can just tell someone that a file is in your public directory on "the ftp server". Microsoft shops and folks that don't need to boot via FTP can probably get away with samba shares.
Doc server. Wiki is probably the best way to go if your team doesn't have to produce overly-formal documentation. Wikis and formal document control systems that I've used have had horrible search, which sucks but I haven't found a great solution. In the past I've done this with a combination of a formal doc tool plus a blog for the informal "howto" kind of notes that are always needed.
Code review. I can't say I've worked anywhere that had a dedicated code review tool. However, when all of the development machines are on NFS so that everyone has a view of your workspace, IM fits the bill reasonably well: post a review request to the chat room with the path to your changes and get an ad hoc on the spot code review. With some discipline and a supportive culture this works very smoothly. (This arrangement could probably be done with an internal pastebin, which I haven't listed here as must-have since I haven't really seen it used.)

What would you add to this list? Any other tools or systems that, if missing, would make you think twice about taking a job?

Use SSH to Forward Multiple Protocols to Multiple Machines

2009-12-04T11:03:00-05:00

(This is part five in a series of posts on ssh.)

Let's say you have a half-dozen machines at work you want to log into. Instead of setting up a remote forwarding connection from each of those machines, you can have the connection from your main machine perform multiple forwardings instead of just one. This even works if some of the machines don't support ssh.

It shouldn't surprise you at this point that you can do this with your config file. On your work machine, you might have something like:

Host tunnel
  HostName cloud.example.com
  User mycloudusername
  IdentityFile ~/.ssh/id_dsa
  Port 22
  RSAAuthentication yes
  PubkeyAuthentication yes
  ExitOnForwardFailure yes
  # tunnel ssh to myworkmachine
  RemoteForward 4022 localhost:22
  # tunnel remote desktop to mywindowsbox via myworkmachine
  RemoteForward 5389 192.168.4.10:3389
  # tunnel http to mywindowsbox via myworkmachine
  RemoteForward 5080 192.168.4.10:80
  # tunnel remote desktop to otherwindowsbox via myworkmachine
  RemoteForward 6389 192.168.4.11:3389
  # tunnel ssh to workserver via myworkmachine
  RemoteForward 7022 192.168.4.2:22

You can add a bunch of forwardings as shown above. Each entry will open the given port on cloud and forward it to the specified port on the specified machine. Now when you run "ssh tunnel" on your work machine, it will connect to cloud and set up the five port forwardings specified in your config file.

Then when logged in to cloud.example.com, you can do, for example, "ssh -p 7022 myserverlogin@localhost" to log into the machine called workserver.

If you mirror the remote forwardings in your home config file as local forwardings, then when you "ssh work" from home you can remote desktop to a windows machine from your home pc by doing "rdesktop -u myworkwinuser localhost:5389" and it will use the tunnel. (The connection will go from your home pc to cloud, to myworkmachine, to mywindowsbox.) The windows machine does not need to know anything about ssh.

Open an SSH Tunnel in Four Seconds or Less

2009-12-03T10:28:00-05:00

(This is part four in a series of posts on ssh.)

As I mentioned in a previous post on ssh configuration, your config file can specify a variety settings for each server.

In fact, the Hosts you use don't even have to exist! (The HostName is the important part.) Consider the following snippet in your ~/.ssh/config.

#
Host work
  HostName localhost
  User myworklogin
  IdentityFile ~/.ssh/id_dsa
  Port 4022
  RSAAuthentication yes
  PubkeyAuthentication yes
  LocalForward 4022 localhost:4022

I'm going to assume remote forwarding is set up and the connection is open from work to cloud as described in this post on remote forwarding; and you've got local forwarding set up from home to cloud as described in this post on local port forwarding.

Now you can do ssh work from your home pc, and it will automatically log you into your work pc with the right credentials using the tunnel on cloud.example.com. And scp simplifies to scp work:/tmp/foo.txt ~/foo.txt -- you don't have to remember the forwarded port numbers.

Typing ssh work is nine keystrokes (eight letters plus enter). If you can type 40 wpm, that's 200 keystrokes per minute, or 3.33 keystrokes per second, which means you can open the tunnel in four seconds!

If you add alias ssw='ssh work' to your ~/.bashrc, you're down to four keystrokes.

Use Local SSH Forwarding to Reduce the Number of Manual Hops

2009-12-02T10:12:00-05:00

(This is part three in a series of posts on ssh.)

Local port forwarding is the same as remote port forwarding but works in the opposite direction. An example is the clearest way to explain.

Assuming you've done the steps in the previous posts, then at home you can run ssh -L 4022:localhost:4022 me@cloud.example.com. This listens on TCP port 4022 on your home machine. Any connections there will be forwarded through the ssh connection to port 4022 on cloud... which, as we recall, gets forwarded to port 22 (ssh) at work. If you leave this connection open, you can run ssh -p 4022 localhost on your home machine and it will connect to work in just one hop. This means that you can use scp to copy files from home to work or vice versa. For example, scp -P 4022 localhost:/tmp/foo.txt ~/foo.txt will copy a file from work to home. (Note: scp needs capital "-P" to give the port. I got it wrong the first time.)

How to Use SSH Remote Port Forwarding to Set Up Secure Tunnels

2009-12-01T09:35:00-05:00

(This is part two in a series of posts on ssh.)

Ssh tunneling can be a bit mind bending at first, but it's simple when you get used to it. Assume that you're trying to ssh between two sites that do not allow incoming ssh. Maybe your IT at work is unenlightened and doesn't have an ssh gateway. And your ISP has braindead configuration rules that don't allow incoming ssh or they make it difficult.

What you need to get around this is a server "in the cloud" that permits ssh logins. This could be a hosting server that you pay for, or even a friend with an enlightened ISP who will give you a login account.

On your work PC, use ssh to login to the "cloud" server. Using the "-R" argument, you tell ssh to listen on a TCP port on the cloud server. Any connection coming in to this server will be forwarded back through the ssh connection to the TCP port you specify. For example, on mymachine.work.com, ssh -R 4022:localhost:22 me@cloud.example.com tells ssh to listen on cloud's port 4022. Incoming connections to that port on cloud will be forwarded to port 22 (ssh) on mymachine.

By default, ssh will only listen to port 4022 on cloud's localhost interface. So to log in to work, you will first need to log into cloud, and then use ssh -p 4022 myworklogin@localhost to log into work.

We'll work around this limitation in the next post in this series.

How to Tell SSH Who You Are

2009-11-30T10:11:00-05:00

Ssh has amazing capabilities that you probably aren't using on a daily basis.

The capability that you probably aren't using, and the easiest to use, is customizing your config file (in ~/.ssh/config) for the various servers you log into.

For example, I frequently log into about ten different servers using at least four different usernames. By default, if I type ssh server the client will use my login name on the client machine to try to log into the server -- which is usually wrong. Instead you can tell your ssh client which username to use on each server. (Thanks to Rich Adams for the tip.)

You can customize a variety of settings -- not just the username. For example, I specify a different identity file for a couple of servers.

This saves a bunch of typing and occasional confusion. (By avoiding login errrors as I try to log into a server using the wrong username and can't figure out why my password isn't working...)

(This is the first post in series of posts about how to get the most out of ssh. Make sure you don't miss the rest of the series: subscribe to my feed.)

One Simple Step for Avoiding Shallow Reviews

2009-11-25T08:55:00-05:00

We've all been guilty of giving a shallow review: "Looks ok."

Given typical defect densities, any non-trivial design or code is going to contain some errors. Even seemingly trivial maintenance fixes are likely to be defective.

It's your job as a reviewer to find as many of these defects as possible. If you're not finding defects, you're wasting your time on reviews.

That "one simple step"? Remind yourself that there are almost certainly defects in the work product you're reviewing, and then find them. It's all about attitude.

If the comments are ugly, the code is ugly

2009-11-16T21:59:00-05:00

If the comments are ugly, the code is ugly (via slashdot). Amen! I get uncomfortable whenever I have to leave a long comment, but it's usually to document some deficiency in a lower layer that the code is working around. Typically broken hardware. (So that someone coming behind me doesn't say, "This is overly complicated, I can simplify it" and then proceed to blow everything up.)

"Good programs do not contain spelling errors or have grammatical mistakes."

Who Else Wants Better Short Term Memory?

2009-11-12T08:13:00-05:00

In "Talent is Overrated", Geoff Colvin at one point discusses how superstars in many fields use the memory technique of "chunking" to boost their short term memory.

His simple example is the 13-letter word "lexicographer". To you and I (assuming you speak English and have a decent vocabulary), it is easy to remember. We don't have to remember 13 letters, we just remember the whole chunk. But when presented with "trgdpxhdewfwm" for 3 seconds, you probably can't remember more than half a dozen letters.

Another example is that chess masters can recall board positions after being shown a chess board with pieces on it for just a few seconds. They do this by chunking sets of pieces together -- almost like "words" -- whereas novices will try to remember individual pieces ("letters").

It struck me that programmers do the same thing when reading and writing code.

The coding standard helps us, by telling us where the chunks are and how to draw the boundaries between chunks.

If you have a coding standard.

And apply it consistently.

When you choose names at random, you destroy your short term memory. You become a crappy programmer, and you drag everyone around you down too.

Quick example in C. Assume you're writing a small module for checking environmental status.

BOOL isOverTemp();
BOOL isUnderTemp();
int GetCurTemperature();
BOOL env_isOverVoltage();
BOOL env_isUnderVoltage();

Anyone writing this pile of gibberish should be fired excommunicated.

Why?

For starters, I just wrote it, and I can't now remember which was camel case and which was underscored. Also: which one was abbreviated?

Before you object that this is a contrived example, two points: (1) yes, it is contrived, that's the point of an example, and (2) I have worked with far worse on an almost daily basis -- the example above is fairly tame.

On the other hand, if you know the coding standard has some simple rules -- and they are followed uniformly -- you can easily remember the function names.

Initial 2-3 letter module prefix ("env" for this example).
All functions are lowercase with underscores.
No abbreviations.

Then we have:

BOOL env_is_over_temperature();
BOOL env_is_under_temperature();
int env_get_cur_temperature();
BOOL env_is_over_voltage();
BOOL env_is_under_voltage();

Now, when you're writing, reviewing, or maintaining code, you don't need to constantly refer to the header or documentation to get it right. A simple three-line coding standard just boosted your memory capacity by over 643%.

(In the interest of full disclosure: I made that number up.)

I realize that implementing even this simple three line standard is controversial because all the camel case folks are pulling out big sticks and the underscore people are grabbing rocks. To which I say: just flip a coin and enjoy the extra brain power. Adopt a three-line standard, follow it, and save the curly-brace debate for the next major coding standard revision.

3 Easy Ways to Stick to a Coding Standard

2009-08-25T13:40:00-04:00

When you're writing python, you don't need a lot of debate over the minutiæ of most coding standards. PEP 8 does that for you. Even better, there are some tools that make it really easy to stick to the standard.

Why do this? Well, for one thing it makes code reviews easier when everyone follows the same conventions. It also makes maintenance easier.

pep8.py is a style checker that enforces the rules of PEP 8. The "official home" (?) at browsershots.org was dead as I was writing this. (Thanks GitHub!) Run pep8.py on your code and it will tell you where you've drifted from the standard.
pylint is lint for python. It provides more functionality than pep8.py, but is not a strict superset. (pep8.py is pickier about whitespace issues.) Pylint is also configurable to enforce various naming rules. Like most lints, the SNR is pretty low, but you can turn off most of the noise and get a reasonable signal for the things you want to check.
Subversion (as well as most other tools) can be configured to run a script every time you check in code. Run one or both of the above tools in the pre-commit hook and bad code will be rejected. (I'd be wary of doing this with pylint unless you've got the categories of "noisy" warnings turned off.)

Hassle Free Way to Kill Sudo'd Jobs

2009-08-12T19:45:00-04:00

Every now and then I have to run a foreground job under sudo that doesn't want to die when I hit ^C. Then it's a hassle to ^Z, get the pid of the sudo job, and sudo kill that pid.

So I wrote a little script (or a template for scripts) that runs the sudo job in the background (but preserves stdout/stderr) and relies on bash to clean up the job when you ^C the script. Only gotcha with this is that you may have to retype your sudo password when you ^C if your authentication has timed out by the time you get around to killing it.

#!/bin/bash
function cleanup()
{
    sudo kill $job_pid
    wait $job_pid
    exit 0
}
trap cleanup SIGTERM
trap cleanup SIGINT
sudo long_running_foreground_process &
job_pid=$!
wait

Using Python's ctypes to Call Into C Libraries

2009-08-05T05:35:00-04:00

The ctypes module makes loading and calling into a dynamic library incredibly easy:

>>> from ctypes import CDLL
>>> libc = CDLL('libc.so.6')
>>> print libc.strlen('abcde')
5

As with everything else in python, it gets even better when you scratch the surface. In the example above, CDLL returns an object that represents the dynamic library. You can access the functions in that library by attribute access ("libc.strlen") or item access ("libc['strlen']"). Both access mechanisms return a callable object.

This callable object has an "errcheck" attribute that can be assigned a callable. We can use this for error-checking our calls into the library. Let's write a simple version of the "kill" command that uses the kill(2) system call.

import sys
from ctypes import *# Load the library.
libc = CDLL('libc.so.6')# Our error checking function. This will receive the
# return value of the library function, the function that
# was called, and the arguments passed to the function as a
# tuple.
def kill_errcheck(retval, func, funcargs):
    '''Check for error -- retval == -1.'''
    if retval < 0:
        raise Exception('kill%s failed' % (funcargs, ))
    return True# Get the kill function from the standard library.
kill = libc.kill# Set the error checker for kill().
kill.errcheck = kill_errcheck# Pass the command line argument as a pid to kill, with
# SIGSEGV (11).
pid = int(sys.argv[1])
kill(pid, 11)

Save this as kill.py. Then, in your shell, try something like this:

# Notice that the 3401 is the pid of the process
# we're putting into the background. Yours will
# be different.
bash$ sleep 120&
[1] 3401
bash$ python kill.py 3401
[1]+ Segmentation Fault         sleep 120
bash$ python kill.py 3401
Traceback (most recent call last):
  File "kill.py", line 17, in
    kill(pid, 11)
  File "kill.py", line 10, in kill_errcheck
    raise Exception('kill%s failed' % (funcargs, ))
Exception: kill(3401, 11) failed

At line 4 of the output we run sleep in the background. At line 5 we learn the pid of this process. At line 6 we run our kill program, giving it the pid we just spawned, and we see the notification from bash that the process was killed (with signal 11, segmentation fault). At line 8 we run our kill program again on pid 3401, but it doesn't exist, the kill system call returns -1, and our error checker raises an exception when it detects the system call failure.

But wait, there's more... I'm working on a follow up post that combines ctypes.Structure with calls into a linux system call.

Yet Another Python Enum Module

2009-07-19T21:21:00-04:00

I didn't like the existing enum recipes, so I cooked up what I feel is a better way of working with enumerations in python. The result is yapyenum, hosted on github. Rather than come up with something new to say about it, I'll just repost the README here:

This module provides named enumerations for python.

Unlike other implementations that I have seen, this is not an anonymous enumeration. To use it derive from the Enumeration class provided in enum.py and list the names of the enum members in the class member enum.

The enum is itself a singleton class. It tries to be immutable via slots and by refusing to allow its members to be changed.

Enum members are an integer subclass with a "name" property and that knows how to pretty print itself. The enum values are interchangeable with integers, which may or may not be what you want.

The class supports membership ("FOO in MyEnum") and mapping a non-instance-member value to a name. (I.e. if FOO uses the integer value 1, then MyEnum.name(1) returns "FOO".)

Tested on 2.4.6, 2.5.2, 2.6.2, on a combination of debian sarge, etch, lenny, and ubuntu 9.04.

There are obvious capabilities that could be added but this does all that I need so far. Patches will be happily accepted.

Permissive license (MIT). Have fun.

For other enum recipes, see:

PEP 354 (rejected enum proposal for python)
The enum module in pypi
Recipe 413486: First Class Enums in Python
Recipe 305271: Enumerated values by name or number
Recipe 67107: Enums for Python
Recipe 81098: python enum with strings
At Stack Overflow: What's the best way to implement an 'enum' in Python?
The Python Cookbook (at page 607 on Google Books)

Several of these recipes have comments along the lines of "if you're using one of these too-fancy enum recipes in python, you're doing something unpythonic". This is likely true, but sometimes unavoidable for social, political, and/or historical reasons.

Python Exception Handling: Cleanup and Reraise

2009-07-15T22:35:00-04:00

I've had this code around for a while and had an opportunity to drag it out the other day and dust it off. The problem: Every now and again there's a situation where you don't really want to catch an exception, but you do want to perform some cleanup and let the exception propagate up the stack. Sometimes there's an extra wrinkle in that the cleanup code may itself throw an exception (that I'm simply going to assume we can ignore).

You can run the file to see the behavior. Simply provide an integer 1-5 as a command line argument and you'll run the selected scenario and see the output. The goal in this case is for cleanup to occur and an exception to be reported as having occurred at line 10.

The code in reraiser1 is wrong because this behaves as if a brand new exception were thrown at line 27. That may not seem so bad, but this code is pretty simple. If this happens and the stack trace is deep, it will be almost impossible to diagnose what went wrong.

The code in reraiser2 shows what happens when a second exception occurs in the except block. A bare raise statement here might be an attempt to re-raise the original exception, but python's rules about re-raising specify that the most recent exception in the scope is what is reraised. In this case, that's the exception thrown from the cleanup function. Again, this makes troubleshooting difficult.

In reraiser3 I worked around the problem in reraiser2 by moving the cleanup function's exception into a separate scope by defining a local cleanup function and calling it from within the except block. This prevents the cleanup function's exception from polluting the scope with an irrelevant exception and we can re-raise the original exception. This results in a stack trace rooted at line 10.

Reraiser4 takes a different approach. Instead of moving the cleanup function's exception into a separate scope, it captures the traceback information from the original exception and then passes it back to the raise statement so that the reported traceback is accurate.

The cleanest way to handle this is to use a finally block as shown in reraiser5. This situation is what "finally" is meant for: it does not trap the exception, it just gives you a chance to clean up before control moves back up the stack to the caller. The presence of finally clues readers in to the fact that you aren't messing with the exception, and that the point of the block is to perform cleanup.

Kindly drop me a note if I've got something wrong above, or if I'm missing a technique (or a common anti-pattern!). Thanks.

Makefiles are Software Too

2009-04-13T22:17:00-04:00

This post was inspired by recent experience with some horrible build scripts from the open source world — but I've seen enough in-house badness over the years that I wanted to establish some basic parameters for build scripts.

This is a really broad way to divide the world, but I see two important categories of build scripts. (By "build script" I mean Makefiles, SConsfiles, Ant files, and their ilk).

Build scripts intended for in-house use, never seen by outsiders.
Build scripts that you distribute to external users.

Both sets of build scripts are important. Let's consider the build script (or Makefile system, or whatever you're using) as a software system independent of the software that is being built. Its primary requirement is to reliably convert your source code into an executable. Any decent build script has numerous other requirements, most of which are probably implied. I have never seen a written requirement for the numerous build scripts I've written. I have only rarely heard them discussed — and never actually posed as "requirements".

I have always taken the following meta-requirements for granted. Based on my experience with some fairly awful build scripts, I guess these aren't universally acknowledged.

The build script should build all the "normal" executables with a single command.
The build script must encapsulate all environment variables within the script. (This is a corollary of the above.)
The build script should not unnecessarily rebuild source code in a directory tree that has previously been built. (This is make's raison d'etre.)
The build script should be documented. It doesn't have to be elaborate, but a five-line comment at the top of the script describing the available command-line variables would be nice.
make clean or its equivalent must work reliably.
It should rarely be necessary to run make clean.
Bonus: the build should be parallelizable, to be able to take advantage of multicore machines and/or distributed builds.

Back to those two categories. If you screw up an in-house build script, it's primarily your team that's going to suffer. I'm not aware of any bugs from SQA or Customers on projects I've been on that were traced back to a build script, but I can see where it could happen. Mainly what I've seen is developers habitually wasting time rebuilding source that should not need to be rebuilt. The worst offender in this respect was a build script that forced a "make clean" at the beginning of every build. In the immortal words of Dave Barry, I'm not making this up. Really.

(Actually, I think it was even worse when I was dealing with a build script that required a "make clean" every time around but didn't have it coded into the script!)

I can't say I have a completely clean conscience in this area: I wrote a big pile of Makefiles for a past employer in which I unknowingly used features (bugs?) of a specific version of GNU Make that were "fixed" in a subsequent version. As time went on it became (a) a much bigger task to fix the Makefile code that used the "feature" and (b) harder to keep the specific version of GNU Make that worked with the Makefiles! I imagine they're still stuck with this situation... sorry guys.

In the second category, when you toss a slapped-together build script to your unsuspecting users you do all of human society a disservice. Now they are not only wasting their time fighting with a bad build script, but they often have the disadvantage of being unable to contact the author directly for support or commiserate with other experienced users of the system.

So do us all a favor: if you don't know what you're doing when you start to write your next build script, please ask for help. Otherwise we'll have to resort to some sort of professional licensing scheme for build script authors.

Five Days to a Django Web App: Day Four, Deployment

2009-03-02T11:48:00-05:00

Thanks for your patience, and for coming back for a discussion of deploying our Django web app.

In case you missed any of the previous posts in this series, here they are:

Day One, Get Ready (Concept and prep)
Day Two, Mockups (Creating a design)
Day Three, Coding (Coding tests, views, templates, and models)

Pre-Deployment

First, we need to make a couple of decisions:

How are we going to push updates to the live site: FTP, git, svn?
How are we going to handle backups?

Version Control as Distribution System

In my case, I'm using svn+ssh to push updates. Notice that this does not require special setup on your server -- you do not need to install the svn stuff that DreamHost or your host may provide. Just do svn init to create a repository on the server (outside the DocumentRoot!). Then point your development pc to svn+ssh://USERNAME@host.example.com/home/USERNAME/svn/PROJECT. (Git works similarly, no support from your host required except the binaries.)

In the directory on the host where you're going to store your project files, point to the same URL. (You could use the file:///.../ url, but I prefer to avoid accessing the repo directly. Superstition?)

Now, whenever you make a change on your development system, just "svn ci" and then on the host "svn up" and restart your fcgi to pick up the new code. Presto! The live site is updated with your change.

Backups

You must have a backup strategy: Your app will have users. Your host's disk will burp. Your users will hate you when the disk burps and you don't have a good backup.

There are probably 374 different ways of backing up your Django app. The two major things you need to capture are the database and your code. If you are storing objects (e.g. uploaded files) outside the database, you'll need to back these up too. The option I'm using is django-backup.

Pull the code from subversion into your project. Rename the directory to "django_backup". Add django_backup to your INSTALLED_APPS. Verify it works by running ./manage.py backup -c. Sanity check the backup by doing zless backups/*.gz. We'll set up a cron job on the host to run this regularly when we deploy.

Deploy

I've previously written about deploying Django apps on DreamHost, so I'm not going to duplicate that here. Keep in mind that you want to use the version control strategy outlined above. Read that article, deploy your app and come back here when you're done.

Post-Deployment

Backup

Let's add that cron job we previously mentioned. On the host, run crontab -e. If you're on DreamHost and this if the first time you've used cron, it will prompt you for an email address to send output to. Then it will dump you into an editor. (Side note: "joe" is the default. If you want something different, like vim, be sure that EDITOR=/usr/bin/vim is set in your environment.)

Set up a job something similar to the following:

MAILTO="YOU@EXAMPLE.com"# m h dom mon dow command 4 2 * * * (cd /home/PATH/TO/PROJECT; ./manage.py backup -c --email=YOU@EXAMPLE.com)

This will run a backup every day at 02:04 (AM). You will get two emails: one with the output from the job, and one with the compressed backup file. (When you get to the point where your database backups are too big for email, you'll need to find another strategy.)

Now we've got another problem to solve: we're going to accumulate a bunch of backups on the disk. Let's get rid of the old backups. This is pretty safe, since we're receiving backup files via email. Add a cron job like this:

14 2 * * * (cd /home/PATH/TO/PROJECT; touch --date=`date --iso --date='10 days ago'` .backup.oldest; find ./backups/ -mindepth 1 \! -newer .backup.oldest -execdir rm '{}' +)

(Formatted here for readability — you need to put that all on one line.)

This will remove backup files older than 10 days. You could do this more concisely with a tool like tmpwatch or tmpreaper, but neither is installed on my host and this incantation should work on pretty much any flavor and installation of linux.

At this point we're deployed and the majority of the work is done. Tomorrow we'll take a look at some maintenance issues.

Firefox Quick Search for Google Maps

2009-02-18T08:43:00-05:00

I recently found myself wanting to quickly figure out how far (both miles and driving time) it is to various destinations. At first I was keeping a Firefox tab open to Google Maps and kept keying in the address for the destination. Then I realized that I could make a "Quick Search" and get the info faster. Add this link to your Quick Searches folder:

http://maps.google.com/maps?f=d&saddr=Elm+St,+ Manchester,+NH+03101&daddr=%s&output=html

[Formatted here for readability. Put it all on one line with no spaces.]

Edit the properties so that the "saddr=" part is your start address. Change the keyword to "gm". This is set to only output simple HTML (no map). If you want the full map, remove the &output=html on the end.

Now, when you need to get a quick idea of how long it is going to take you to get somewhere, open a new tab and type "gm city, state".

Five Days to a Django Web App: Day Three, Coding

2009-02-11T10:14:00-05:00

Thanks for coming back for Day Three!

[Note: Sorry this post is a day late. It was all ready to go late yesterday, but some of the code included below triggered a bug either in WordPress or ScribeFire and the whole post got mangled. I managed to resurrect it today from drafts, and I think it's coherent, but if you find some problem with it please drop me a note.]

Progress So Far

Yesterday we built some mockups, just HTML and CSS — nothing active. Hopefully you’ve had a chance to run your mockups past a couple of people for feedback and ideas.

Armed with these mockups, we’re ready to get started coding. Today’s post is the longest in the series. Take it in chunks if you need to. (I didn’t write it all at once either.)

Foundation

MySQL

First, go to your web host’s control panel and create two MySQL databases for this project. I created "resumandb" and "test_resumandb". This latter database is needed for running the built-in test system.

You’ll get an error/warning if the test database exists when you first run tests. However, the control panel needs to set user privileges and it seems like the only way it knows how to do this is by creating the database.

Set up the same databases on your local system: bash$ mysql -u root Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1969 Server version: 5.0.32-Debian_7etch8-log Debian etch distributionType 'help;' or '\h' for help. Type '\c' to clear the buffer.mysql> CREATE DATABASE resumandb; Query OK, 1 row affected (0.00 sec)mysql> GRANT ALL ON resumandb.* TO 'resuman'@'localhost' identified by 'reallY_baD_9passworD!'; Query OK, 0 rows affected (0.11 sec)mysql> GRANT ALL ON test_resumandb.* TO 'resuman'@'localhost' identified by 'reallY_baD_9passworD!'; Query OK, 0 rows affected (0.0 sec) Notice that you don't need to create the test database. As mentioned above, this will happen when you run tests. ### Project Skeleton What follows is a bunch of steps: do-this, do-that. At the end of this section you should have a runnable (but empty) project that is ready to start hanging functionality onto.

Then generate a skeleton for your project: django-admin.py startproject YOURPROJECT.

Then dive right in to settings.py and change a few things:

Set the admin email to your email address.
Set the MySQL database according to your web host's settings.
Timezone.
MEDIA_ROOT, MEDIA_URL and variants — see note below.
TEMPLATE_DIRS — see note below.

To make it easier to use the same settings file to test both locally and on your host, I add the following to my settings.py to set MEDIA_ROOT and TEMPLATE_DIRS:

import osdef full_path_to(path):
    '''This makes this settings file relocatable.'''
    return os.path.join(os.getcwd(), path)MEDIA_ROOT = full_path_to('static/')
TEMPLATE_DIRS = (
    full_path_to('templates')
)

Then you just have to make sure to cd to the directory where your settings.py lives whenever the app runs (more on this when we deploy to the host). I find this easier than monkeying with PYTHONPATH.

Similarly, the following will change your URLs based on whether you're running locally or on the host. (Just make sure you test for a directory that only exists locally! My paths are a little different on the host.)

MEDIA_URL = 'http://media.resuman.com/resuman/static/'
if os.path.exists('/home/brian/projects/resuman'):
    MEDIA_URL = 'http://localhost/apache2-default/resuman-static/'ADMIN_MEDIA_PREFIX = 'http://media.example.com/admin_media/'
if os.path.exists('/home/brian/projects/resuman'):
    ADMIN_MEDIA_PREFIX = 'http://localhost/apache2-default/resuman-admin/'

Now it's time to generate the app: ./manage.py startapp YOURAPP. Notice that the app name should be different from the project name (otherwise it gets too confusing later on).

Edit your YOURPROJECT/urls.py to include a reference to YOURAPP:

    (r'^funnel/$', include('resuman.jobfunnel.urls')),

Also uncomment the admin lines so you can use the built-in admin app.

Now it's time to generate the app: ./manage.py startapp YOURAPP.

Now edit YOURAPP/urls.py -- paste in the needed bits from the existing urls.py, but drop the admin lines and the include().

In your settings file, add the admin and admindoc apps and "YOURPROJECT.YOURAPP" to INSTALLED_APPS.

Under the YOURPROJECT directory, create a templates directory and a static directory. Copy your HTML files from the mockup to the templates directory. Copy the CSS file to the static directory.

My system is configured by default to serve from /var/www/apache2-default/, and we specified above to fetch static files from http://localhost/resuman-static/, so we need to do the following to make this possible (substituting your paths, of course): bash$ ln -s /home/brian/projects/resuman/static /var/www/apache2-default/resuman-static bash$ ln -s /home/brian/projects/django/git/django/contrib/admin/media/ /var/www/apache2-default/resuman-admin Restart apache.

Change directory to YOURPROJECT and ./manage.py syncdb; ./manage.py runserver.

Browse to http://127.0.0.1:8000/admin/. Log in, and you should see the admin app in all its glory. If the stylesheet didn't load (ie. it looks really ugly), View Source on the page. At the top, find the URL to that ends something like .../resuman-admin/css/base.css. Copy-paste this into the address bar. It probably doesn't load. Verify that it is the right URL — if not then change your settings.py to have the right URL base. If the URL is right, then you need to fix your symlink, server config, or permissions (I often get bit by having the wrong permissions).

Adding Some Meat

This wouldn't be a bad time to push a snapshot into your version control system (e.g. git init; git add .; git commit -m'YOURPROJECT skeleton done').

Now we're finally ready to write the first view for this app. Edit YOURAPP/urls.py. Lay out the URL map that you want to use for your app. We're on a tight five day schedule, so don't go nuts! There's only time to get a couple of pages done. Don't worry, you can add more later. Here's what mine looks like:

from django.conf.urls.defaults import *urlpatterns = patterns(
    '',
    (r'^$', 'resuman.jobfunnel.views.dashboard'),
    (r'^add/$', 'resuman.jobfunnel.views.add_job'),
    (r'^edit/$', 'resuman.jobfunnel.views.edit_job'),
)

Remember that my toplevel urls.py includes this based on the ^funnel/$ pattern, so each of the patterns above will have …funnel/ as a prefix in the URL.

Now let’s write our first couple of tests. Edit YOURAPP/tests.py and add something similar to the code below. You’ll have to change URLs and logins. The class ViewTestCase is defined in viewtestcase.py, a convenience TestCase subclass I wrote for testing Django views. Copy that file into YOURAPPNAME directory.

from viewtestcase import ViewTestCase
class DashboardViewTestBase(ViewTestCase):
    # Override in subclass to use post/head/etc. Must match a
    # method defined in django.test.TestCase.
    TESTMETHOD = 'get'    # Override.
    TESTURL = '/funnel/'
    TESTARGS = {}
    TEMPLATE = 'dashboard.html'class DashboardViewLoginTest(DashboardViewTestBase):
    # We're expecting an error, so set TEMPLATE to None to avoid getting a bogus test failure.
    TEMPLATE = None    def test_login_required(self):
        """
        Tests that a login is required to view the page.
        """
        self.expect_login_redirect()
        returnclass DashboardViewTest(DashboardViewTestBase):
    # This uses TEMPLATE from the parent class.    # Set username and password and the base class will automagically login the test client.
    USERNAME = 'brian'
    TESTLOGIN = (USERNAME, 'a')    def test_logged_in_ok(self):
        pass

Now run the test: ./manage.py test. You should see exactly two failures. If a bunch of stuff fails (like built-in django tests), fix whatever is wrong before continuing.

bash$ ./manage.py test
Creating test database...
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table jobfunnel_job
Installing index for admin.LogEntry model
Installing index for auth.Permission model
Installing index for auth.Message model
Installing index for jobfunnel.Job model
Installing json fixture 'initial_data' from '/home/brian/projects/resuman/../resuman/jobfunnel/fixtures'.
Installed 1 object(s) from 1 fixture(s)
..........EE......
======================================================================
ERROR: test_login_required (resuman.jobfunnel.tests.DashboardViewLoginTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/brian/projects/resuman/../resuman/jobfunnel/viewtestcase.py", line 98, in setUp
    self.fetch_view(self.TESTMETHOD, self.TESTURL, self.TESTARGS)
  File "/home/brian/projects/resuman/../resuman/jobfunnel/viewtestcase.py", line 79, in fetch_view
    self.response = function(testurl, testargs, **extra)
  File "/usr/lib/python2.5/site-packages/django/test/client.py", line 277, in get
    return self.request(**r)
  File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py", line 77, in get_response
    request.path_info)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 183, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 183, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 124, in resolve
    return self.callback, args, kwargs
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 136, in _get_callback
    raise ViewDoesNotExist, "Tried %s in module %s. Error was: %s" % (func_name, mod_name, str(e))
ViewDoesNotExist: Tried dashboard in module resuman.jobfunnel.views. Error was: 'module' object has no attribute 'dashboard'======================================================================
ERROR: test_logged_in_ok (resuman.jobfunnel.tests.DashboardViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/brian/projects/resuman/../resuman/jobfunnel/viewtestcase.py", line 98, in setUp
    self.fetch_view(self.TESTMETHOD, self.TESTURL, self.TESTARGS)
  File "/home/brian/projects/resuman/../resuman/jobfunnel/viewtestcase.py", line 79, in fetch_view
    self.response = function(testurl, testargs, **extra)
  File "/usr/lib/python2.5/site-packages/django/test/client.py", line 277, in get
    return self.request(**r)
  File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py", line 77, in get_response
    request.path_info)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 183, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 183, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 124, in resolve
    return self.callback, args, kwargs
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 136, in _get_callback
    raise ViewDoesNotExist, "Tried %s in module %s. Error was: %s" % (func_name, mod_name, str(e))
ViewDoesNotExist: Tried dashboard in module resuman.jobfunnel.views. Error was: 'module' object has no attribute 'dashboard'----------------------------------------------------------------------
Ran 18 tests in 4.364sFAILED (errors=2)
Destroying test database...

Let’s write the view so the test will pass. Edit views.py:

@login_required
def dashboard(request):
    return render_to_response('dashboard.html',
                              {'title': 'Dashboard'},
                              context_instance=RequestContext(request))

This view uses a template called dashboard.html. Let’s make that template. Go back to the mockup for the dashboard. Copy everything into templates/base.html, then rip out the content so all you have left is the generic skeleton of a page in your app. Something like this (notice that this is using the title variable passed into the context by the view):

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><!--
# Resuman: keep track of job applications, cover letters, and resumes
#
# Copyright (c) 2009, Blakita Software LLC
# All rights reserved.
--><html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />  <title>
    {{ title }}
  </title>  <style type="text/css">
    @import "{{ MEDIA_URL }}base.css";
  </style>  {% block scripts %}
  {% endblock scripts %}</head><body><div id="container"><div id="menu">
TBD
</div> <!-- end div=menu --><div id="header">
<h1>{{ title }}</h1>
</div> <!-- end div=header --><div id="content">{% block content %}
{% endblock content %}</div> <!-- end div=content --><div id="footer">
<div class="nav">
  <ul>
    <li><a href="/">Home</a></li>
    <li><a href="/blog/about">About</a></li>
    <li><a href="/privacy.html">Privacy</a></li>
  </ul>
</div><div class="copyright">Copyright © 2009, Blakita Software LLC</div>
</div> <!-- end div=footer --></div> <!-- end div=container --></body>
</html>

Now take the content you ripped out and put it into dashboard.html:

{% extends "base.html" %}{% block content %}
<ul id="funnel">
  <li class="phase first">
    <div class="phase">Applied</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">AAA</li>
    </ul>
    </div>
  </li>
  <li class="phase">
    <div class="phase">Confirmed</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">BBB</li>
        <li class="company">CCC</li>
    </ul>
    </div>
  </li>
  <li class="phase">
    <div class="phase">Screen</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">Fubar</li>
    </ul>
    </div>
  </li>
  <li class="phase">
    <div class="phase">Interview</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">Rabuf</li>
    </ul>
    </div>
  </li>
  <li class="phase">
    <div class="phase">Offer</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">Oof Rab</li>
    </ul>
    </div>
  </li>
  <li class="phase last">
    <div class="phase">Start!</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">Rab Oof</li>
    </ul>
    </div>
  </li>
</ul>{% endblock content %}

Obviously this still just has dummy static data. We'll get some active data in there very soon, but now we're ready to rerun the test. It should pass this time. If not, fix the problem. When you get all tests passing, celebrate!

For real data, we need to write the model(s) used by this app. Let's add another test:

class DashboardViewTest(DashboardViewTestBase):
    USERNAME = 'brian'
    TESTLOGIN = (USERNAME, 'a')    def test_company_list(self):
        self.expect_div_content('content', 'Foobar Corp')
        return

Edit models.py to add the model:

from django.contrib.auth.models import User
from django.db import modelsclass Job(models.Model):
    applicant = models.ForeignKey(User)
    company = models.CharField(max_length=80)

The test depends on having a job in the database. Let's set up a test fixture. Edit YOURAPPNAME/fixtures/initial_data.json:

[{"pk": 1, "model": "auth.user", "fields":
    {"username": "brian", "first_name": "", "last_name": "",
     "is_active": 1, "is_superuser": 0, "is_staff": 0,
     "last_login": "2009-02-06 13:50:02", "groups": [], "user_permissions": [],
     "password": "sha1$0fcc8$c4cf5184f5c005f90165e782fb090e7d75b72986",
     "email": "brian@example.com", "date_joined": "2009-02-06 13:44:04"}},
 {"pk": 2, "model": "auth.user", "fields":
    {"username": "alan", "first_name": "", "last_name": "",
     "is_active": 1, "is_superuser": 0, "is_staff": 0,
     "last_login": "2009-02-06 13:50:02", "groups": [], "user_permissions": [],
     "password": "sha1$0fcc8$c4cf5184f5c005f90165e782fb090e7d75b72986",
     "email": "alan@example.org", "date_joined": "2009-02-06 13:44:04"}},
 {"pk": 1, "model": "jobfunnel.job", "fields":
    {"position_url": "http://example.com/career/", "title": "Foobar Eng",
     "company_url": "http://example.com/", "company": "Foobar Corp", "applicant": 1,
     "phase": "Apply", "date": "2009-02-11", "position": "Engineer",
     "notes": "Applied via website"}}
]

This will populate the database with a couple of users, both with password "a", and a job before each test runs.

Run this test, expecting exactly one failure — the content div does not contain the expected string. Let's grab the list of jobs in the view and pass it into the template:

@login_required
def dashboard(request):
    jobs = models.Job.objects.all()
    return render_to_response('dashboard.html',
                              {'title': 'Dashboard',
                               'jobs': jobs,
                               },
                              context_instance=RequestContext(request))

I won't paste all of the code here again, but we need to edit the template to use the jobs list:

    <ul class="companies">
      {% for job in jobs %}
        <li class="company">{{ job.company }}</li>
      {% empty %}
        <li class="company">No jobs in this phase.</li>
      {% endfor %}
    </ul>

Now rerun the test and expect it to pass. Hooray!

One last refinement before we quit for today: users shouldn't be able to see each others applications. The way this is coded, all jobs are going to show up on everybody's dashboards. Not good. Here's another test that checks that the application for Foobar Corp only shows up on Brian's dashboard, not on Alan's.

class PrivateDashboardViewTest(DashboardViewTestBase): USERNAME = 'alan' TESTLOGIN = (USERNAME, 'a') def test_private_applications(self): assert('Foobar' not in self.get_div_content('content')) return

Run the test, watch it fail, and then change one line in the view:

jobs = models.Job.objects.filter(applicant=request.user)

Now the all the tests should pass.

Push a copy of your code into your version control tool. Take a break, you deserve it.

For "homework", put together your other views in the same way as this one. We’ll look at deployment tomorrow.

Five Days to a Django Web App: Day Two, Mockups

2009-02-10T09:40:00-05:00

Welcome back.

Ready for Day Two? Did you get your "hello world" app running on your host?

Where We Stand

Yesterday we nailed down our concept, bought a domain name and hosting, set up our toolkit, and deployed a practice app on the host.

Here's what I'm going to build: a web app to keep track of job applications, cover letters, and resumes. The working name for the project is "resuman". Unfortunately, that obvious domain name was already registered and I'm not up for buying from the current owner. So after trying a bunch of more-or-less obvious combinations, I grabbed "yresu.me", and I'll deploy the app at "trackm.yresu.me".

Notes on Domain Registration

It's worth making a couple of notes here on the domain registration process. I use GoDaddy for my registrations. In the past I've done registration with my preferred host, but if you end up switching hosts it's a hassle to move the domain registrations. Another complication: hosting services like DreamHost only support registrations on the major TLDs (.com, .org, .net, .info), while a full-service registrar like GoDaddy supports alternatives like .me.

Lastly, I like the bulk domain search service from Dotster. You can search for up to 50 names at a time. So if your first choice is unavailable, use Dotster's bulk search tool to enter a whole mess of variations and see what's available. This is the process I used to find yresu.me — the search included resu.me, esu.me, su.me, and myresu.me, all of which were taken.

Make Some Mockups

Before we commit any design ideas to HTML, let's take some great advice from 37signals: sketch a bunch of drafts on paper first. To back up this idea, check out this story posted by Jeff Atwood about quantity vs. quality:

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the "quantity" group: fifty pound of pots rated an "A", forty pounds a "B", and so on. Those being graded on "quality", however, needed to produce only one pot - albeit a perfect one - to get an "A".

Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the "quantity" group was busily churning out piles of work - and learning from their mistakes - the "quality" group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.

So my advice to you is to produce no fewer than five different designs on paper. It only takes a few minutes to sketch something out and see what it looks like. I did a handful of drawings on paper with my kids' crayons, and then some refinements on my whiteboard once I had chosen the general design. So use whatever media you have handy, and don't worry about getting it perfect. Run your sketches by a couple of innocent bystanders for some feedback.

Once you've got a good design on paper, make mockups in HTML and CSS. Steve Dennis at subcide.com has a great walkthrough on creating a CSS layout from scratch. If you don't have an established process for building up a design, follow his tutorial. Putting together the HTML+CSS takes a little more time than sketches on paper, but it's still worth doing a couple of different designs to see what grabs you. I worked up two different CSS layouts on top of the same HTML. Run these designs by some people — now that you've got some code, you can upload it to your site and email a link to some friends. Pick the better of the two (or three).

Better yet, upload the mockups to your host and leave a comment here with a link to them. Traffic here is low enough that I should be able to reply with feedback. (Although feel free to circulate a link to this series to a hundred of your friends, or post it to digg. I'll at least reply to the first 50 comments...)

Tomorrow we'll start writing some code.

Five Days to a Django Web App: Day One, Get Ready

2009-02-09T10:15:00-05:00

This is the first in a series of posts that will walk through the steps of designing, building, and deploying a complete web app using Django. I'm going to assume you know the basics when it comes to Python, Django, HTML, CSS, Javascript and some basic tools. This will be a part time effort, so you should be able to work alongside me in a couple of hours a day.

A word about costs: you'll need to spend ten bucks or so for a domain name, and anywhere from $30 to $150+ for hosting if you don't already have these set up. Of course, you can just follow along on your server at home for free, but you'll learn more if it's all live when everything is said and done.

Let's get started.

Day 1: Get Ready

Concept

We need to figure out what we'll be building, and we need to get a few things set up.

First, you need an idea. It doesn't have to be the next Twitter. Try to pick something simple that you'll be able to code up in a couple of evenings, but useful enough that it's worth doing right.

Let that simmer on the back burner, and let's get some stuff set up.

Hosting

You will need someplace to host the app. As noted above, you can do this on your own machine at home, but it's good practice to set it up live — the learning will be stickier too.

Webfaction is highly recommended for Django hosting, but it can be a bit pricey if you're just experimenting.

Deploying Django on Dreamhost will suffer a bit performance-wise, but if you google for a coupon you may be able to get up to $97 off. So for $30 you can get a year's worth of hosting. Not bad.

Go off and do this. Now. This post will still be here when you come back.

Domain

Obviously, you're going to need a domain name where your app will live.

How's that idea you've been simmering for the past 20 minutes? Good, now pick a domain name to go with it. You can let this simmer for a while too, but remember that the name can take a day or two to propagate through DNS once you've purchased it.

Not sure about Webfaction, but you can buy this through Dreamhost for $10 or $15. Depending on what kinds of promotions they're offering, you may even have a credit for a free domain name when you signed up for hosting.

Go ahead and snag that domain now, I'll wait patiently...

Tools

Since you don't want to develop on the hosting platform, you'll need to install some things on your own machine so you can develop and test. This is the minimum I've been using, please leave a comment if there's something you think I'm missing.

Apache with mod_python (used at Webfaction) or mod_fastcgi (used at Dreamhost). If you have a different host (or you're reading this at some future date and the options have changed), just make sure that what you're using matches them.
Editor
Firefox with the Firebug and Web Developer add-ons.
Python
Django. I use the trunk. (Actually, I pull from my github fork.)
Selenium

It will be helpful if you can install the tools in the same way — including the same paths — that you have on your web host. If this isn't practical, it isn't a big deal, but it will make your life easier down the road when you're trying to figure out why something works locally but doesn't work on the live site.

That's it for today. As homework, set up a practice "hello world" app on your host so that you know everything is working.

Update: continue this series with day two: mockups.

Subscribe to The Daily Build to make sure you get all the posts in this series!

Jesse Noller on Python

2009-02-09T08:12:00-05:00

Jesse Noller has been republishing articles he wrote for Python Magazine. These are very informative: I learned new things about the "with" statement in python 2.5 and tonight I was introduced to Paramiko, a library for using SSH.

The Toolkit of a Software Engineer

2009-02-05T16:48:00-05:00

This is a rundown of the things you should have in your toolkit -- doesn't matter whether you call yourself a software engineer, programmer, developer, code monkey, etc.

Editor

You must have an excellent editor. It should slice, dice, puree and mince, all with minimal effort. Seriously, it should have support for searching across files, "tags" (jumping to the definition of an identifier), ideally provide tooltips for function calls while you're writing, automatic template generation, and pushbutton compilation that can step you through the list of compile errors, if any.

Pick a good editor, and become an expert at using it. It's worth the investment.

Toolchain (Compiler)

I can't really say compiler any more, since many of the languages we're using aren't compiled in the traditional sense. So instead I'll call this the toolchain. For your chosen language, you need to choose the most appropriate toolchain. It may sometimes seem like you don't have much choice: there's only one python, right? Wrong. You have the choice of targeting different platforms (CPython, Jython or IronPython), and also of choosing the version you'll use. You may need to choose an older version so that you can match what's installed on your hosting provider. You may even need to keep around multiple versions — maybe you're a library writer and need to ensure portability across various versions.

Things that are part of your toolchain: compiler or interpreter and runtime. In other words, whatever it takes to make the software run on your customers' computer.

Version Control

AKA VC, software configuration management (SCM), source control. This is a tool that keeps track of the various versions of your software and allows you to work effectively with other engineers. There are a number of different VC tools, with a wide variety of features, complexity and costs.

If you haven't used VC before, take a look at subversion. It's free (open source), widely used, easy to install and use, and stable. There are other tools that may have better technical merits, especially for certain environments, but I think subversion is the best place to start from an ease-of-learning perspective.

Build

For your software to have value for your customers, you need to be able to convert program text into executable software. If your software is a one-file script, maybe all you need to do is publish it to your website. In this is the case, you're allowed a pass on having a build tool, but you should still consider an automated solution.

If you have to deliver something more complex, an automated build tool is required.

Even if you're just publishing a handful of scripts to your website as a tar or zip file, you still need to perform the following: gather the scripts, tar them, name the tarball something sensible, and upload. Do this by hand, and eventually you will skip a step and your customers will end up downloading an unusable mess.

At the opposite extreme, you may have hundreds of files that need to be compiled into a handful of libraries and then linked into an executable. You may have dozens of dependencies on external libraries. Translations to other natural languages. All of this needs to be portable across a dozen platforms.

This is why "make" was invented, and why other tools with similar goals continue to evolve to solve these complex problems. Ant, Maven, Scons, an endless parade of Make variants, etc. Pick the build tool that best works with your toolchain and specific situation.

Review

You need a way to review your work. If you're working alone, it may be enough to simply review your work in your editor — though I'd recommend at least printing it out. For whatever reason, it seems harder / less effective to review a work product in the medium that produced it. A better alternative may be to use the view / diff functionality in your VC tool.

If you're working with others, it will be helpful to install a code review tool that helps package your work product, distribute it to your peers for review, and manage feedback.

Test

Some languages come with support for unit test built in — python includes it in the standard library. You can find a handful of different libraries for c++, and for seemingly any other programming language known to man.

Your testing needs may also require a framework like selenium, which allows automated testing in a web browser.

If you're doing embedded software, you're probably going to have to jump through a bunch of hoops to automate your testing. Unfortunately, you'll have to figure out how big the hoops are, how many you need, how far off the ground they need to be, whether they should be electrified, and then you'll have to build them yourself. Annoying, but very much worth it in the end.

Continuous Integration / Automated Build

Again: if you're working alone you can skip this, though I still think it's useful. To be clear, "continuous integration" is more than a tool. It's a way of life. However, this post is about your toolkit, and CI is a very useful category of tool. There are a number of different CI tools available, each with a slightly different focus and set of features. Pick the tool that has the features you need, connect it to your VC system, and enjoy the automation.

Lint

For those of you who aren't familiar, lint is an old-school tool for finding potential trouble spots in C code. There are many tools available for C and other programming languages that help to sniff out trouble spots in code. Some tools can even spot design anti-patterns. C has lint. Python has pylint. Coverity Prevent is a commercial tool (expensive) that can find deep bugs in C, C++ and Java.

I don't want to advocate sloppy programming practices: you shouldn't need to run lint on your code. But if there's a tool available that can effectively remove defects from my software, I'm going to use it.

("Effective" defined to mean costing less to remove defects than some alternative, which is often either system testing or "customer testing". Before you go spending tons of money on an expensive lint, are you doing code reviews? This is an extremely cost effective practice. Spend your time and money there first.)

I'll continue this in another post. Meanwhile, what's in your toolkit? Favorites from the categories above? Categories of software tools that you want to make sure I don't miss?

Deploying Django Apps on Dreamhost

2009-02-03T10:42:00-05:00

The Dreamhost wiki article on Django helped, but all the steps starting from scratch aren't really documented in one place. Hopefully the list below will help, but since I'm writing it after the fact and I had to go through a couple of iterations to get it right, there are probably some things that aren't 100% right.

Read the python article. Set up virtualenv into $HOME/local.
If you've already messed around with installing MySQLdb and/or other packages, remove them and start over.
Install ez_setup (easy_install) as describe in the article.
Install the MySQL egg: easy_install MySQL_python
Follow the setup steps in the Django article. Do the "myproject" test using sqlite3. Really. It helped me find a couple of things I was doing wrong with my real (i.e. more complicated) MySQL project.
As a deviation from the Django setup instructions, I prefer to use a fork of the Django codebase with some of my own patches. If you're in the same situation, use github. I forked the github "unofficial copy" of the subversion code, added a couple of patches that aren't in the trunk yet and a couple of my own, and cloned a copy into my dreamhost account.

Five Things That Do Not Belong In A Review Checklist

2009-01-27T17:09:00-05:00

This is the second half of an article I posted about using a checklist to prevent security errors. There, I said that you have 15 checklist items max, and you shouldn't waste any of those questions on silly things like "Does the code follow the coding standard?".

Jason Cohen pointed to an article of his in which he said "List no items that can be automated." (His emphasis, but I second the motion.)

In the context of the SANS paper, let's look at five items that should be automated so that no human has to find the defects:

CWE-665: Improper Initialization. As the write-up for CWE-665 suggests, you could use a programming language that forces you to explicitly initialize all variables before use. Or, if you're stuck with something like C, make sure you turn on the warnings in your compiler, or use a static analysis tool (e.g. lint) to verify that all variables are initialized before being used. Tools like Coverity can be very sophisticated in their static analysis.
CWE-119: Failure to Constrain Operations within the Bounds of a Memory Buffer. Same goes for this one. First, use a programming language that doesn't require attention to every tiny little detail of string handling. Failing that, apply static analysis. It's also worth performing runtime analysis (e.g. electric fence, Purify) with appropriate test cases to verify that you've avoided buffer overflows.
CWE-404: Improper Resource Shutdown or Release. Even garbage collected languages can have problems with resource release. First, some GC systems have problems with circular references. Second, GC systems are typically worried about releasing memory. You also have to worry about database handles, file handles, and sockets. Configure your static analysis tool to track resources like these so that you don't have to do it manually.
The write-up for CWE-404 also mentions in passing that you should "wash your garbage before you dispose of it". This is useful for two reasons: first, an attacker won't have access to the contents of previously used memory, and second, it often makes debugging memory problems easier if you write a known value into freed memory. (I modified malloc() at my last job to write 0xcacacaca into freed memory so that we would know right away when someone stepped on a turd, um, I mean used freed memory.) Configure your libraries to shred objects before you free them and you won't have to worry about it on a line-by-line basis in the code you review.
CWE-362: Race Condition. I'm typically worried less about security than the nightmare of isolating and debugging race conditions. I haven't seen any tools that detect race conditions well, but Coverity does a decent job of telling you when you've mishandled a race condition in a couple of cases: first, it warns you when you fail to release a lock, and second, it warns you when you access a resource that is statistically accessed from within a lock. So you still have to beware of race conditions (at the design level is the best place to find these), but there is an option for finding tedious errors in the mechanics of dealing with races.

And now I'm going to throw out everything above. Remember that the tools are built to detect common errors. If it is really important that you find a certain class of bug, put it on your checklist. In fact, if it's super critical, make it a checklist of length one. For example, I've gone over codebases with the single-minded focus of finding race conditions. When you perform a review like this, it's amazing how many defects you might find in areas that you never suspected.

Seeking Timelog Beta Testers

2009-01-27T14:09:00-05:00

Nutria is an online timelog. I'm hoping to start a live beta this week, initially light on features but with more to follow. I need some beta testers to give it a spin and provide feedback. Sign up to test and you will have my undying gratitude. Thanks!

How To Use A Checklist to Prevent Security Errors

2009-01-23T12:23:00-05:00

The 2009 CWE/SANS Top 25 Most Dangerous Programming Errors has been out for a while now. Maybe you've already eliminated all of these errors from your code. In case you haven't, this post will help you develop a checklist that you can use to eliminate these errors starting at the architecture level and moving through design, code, and testing. Before we get to the security aspect, first let's take a quick detour through the mechanics of a review checklist.

Why A Review Checklist is Essential

Most (all?) experts recommend the use of a checklist for performing design and code reviews. It's more effective to ask a set of questions about the item being reviewed than to simply stare at a bunch of code looking for problems. The checklist focuses your attention on aspects that are most important.

If you want to perform a broader review, develop two or three checklists with different foci and give the lists to separate reviewers. The defect lists from each reviewer will have less overlap than if you gave everyone the same list.

What Your Review Checklist Should Look Like

The following advice is partly based on advice from "Software Inspection" by Gilb and Graham. A good checklist is a list of questions. For practical purposes, I'm strongly in favor of a checklist that does not exceed one side of a printed page — at normal font size. For me this means a checklist of about 15 items, 20 as an absolute maximum. Remember, we're focusing, which means looking harder for just a few types of items. If you're using online checklists, I'd extend this to mean that it should fit on the screen without scrolling. Either way, I don't think you should have more than 20 questions or so at the max. Fewer is probably better, more is too overwhelming.

Checklist items are yes/no questions. They should all be phrased such that a "no" answer means something is broken.

When reviewing, proceed through the entire item under review (the entire code listing or design) with each question. Don't work linearly, this is a horrible way to review code. I prefer to review on paper, simply because I can make little notes and marks all over the paper as I go through it. For the first question, you might work mostly front-to-back. Then the second question might send you backwards through the code. After a couple of passes over the code you have some familiarity with it, so the third question might send you straight to a location that has potential problems.

How To Choose Questions for the Review Checklist

Since the whole point of the checklist (and reviews in general) is to find defects, the questions should focus on finding the highest value defects. Why waste valuable checklist real estate on a question like "Does the code conform to formatting conventions?" Duh. If your team has a formatting convention, and you look at a piece of code that doesn't conform, it's going to jump off the page and bite you in the nose. You don't need a checklist question for this!

Back to the specific problem at hand: security errors. SANS has done a great job of sifting through tons of security errors and boiling them down to the top 25. We could probably develop a 100 question review checklist from their Top 25 list, but it would be too big to be usable. So let's focus on the most important for our application.

For this exercise, our application is a blog like Wordpress but much simpler: no plugins, no comments, single author. This is not a hosted application like Blogger, so we are less concerned about vulnerabilities like cross-site scripting, SQL injection, and OS command injection. We do need to make sure that only the author is allowed to post, so let's look at the "Porous Defenses" category.

CWE-285: Improper Access Control (Authorization)
CWE-327: Use of a Broken or Risky Cryptographic Algorithm
CWE-259: Hard-Coded Password
CWE-732: Insecure Permission Assignment for Critical Resource
CWE-330: Use of Insufficiently Random Values
CWE-250: Execution with Unnecessary Privileges
CWE-602: Client-Side Enforcement of Server-Side Security

The entry for CWE-285: Improper Access Control lists the following for prevention and mitigation:

For web applications, make sure that the access control mechanism is enforced correctly at the server side on every page. Users should not be able to access any information that they are not authorized for by simply requesting direct access to that page. One way to do this is to ensure that all pages containing sensitive information are not cached, and that all such pages restrict access to requests that are accompanied by an active and authenticated session token associated with a user who has the required permissions to access that page.

Here are some straightforward checklist questions we can extract from that description:

Does every administrative page enforce access control on the server side?
Are unauthenticated (anonymous) users prevented from accessing administrative pages?
Is caching disabled for administrative pages?
Does every administrative page validate the session token properly?

Now you can go walk through your design and/or code to verify that you can say "yes" to each of these questions.

If you found this post helpful, subscribe to The Daily Build to get updates as they are posted. Thanks!

"2009 CWE/SANS Top 25 Most Dangerous Programming Errors"

Today's Time Log Is Tomorrow's Historical Data

2009-01-20T08:16:00-05:00

The only good way to create any kind of a reliable estimate is by using (your own) historical data.

If you are trying to create an estimate for a new project and are having a hard time, don't lament your lack of historical data. The best thing you can do is to set up a system for tracking time on your projects. Today's time log is tomorrow's historical data. (When I say "you" I mean either you, singular, if you work alone or you, plural, to refer to your project team.) Do the best you can on this estimate, recognize that it's probably a bunch of nonsense, and move forward.

Before you start working on the project you just (mis-)estimated, plan how you're going to track time spent on it. This doesn't mean you have to establish the Time Tracking Committee, hold a dozen meetings every day, and publish a seventy-page document describing how time will be tracked. Just pick a solution and spitball it past your team. Lightweight is best for a first pass, you'll be able to get a feel for what works and what doesn't in your situation.

Some things to keep in mind:

You might work a nominal eight hour day, but you probably spend less than three actually working on your project. Meetings, support for previous projects, and the water cooler suck up a lot more time than you realize.
Some meetings are directly project-related (design review, etc) and should probably be charged to the project. Oooh, I said "charged". Don't get too uptight about applying all of your time to one bucket or another. Keep it lightweight and easy to implement and you'll increase the likelihood that you'll be able to stick with it.
Spreadsheets can be a convenient way to track time for individuals or small teams. If a team is going to use a spreadsheet-based solution, take a look at Google Docs and set up a shared spreadsheet.

At the end of the project, you can compare the actual results to the estimate you made. Disappointing, eh? Don't worry, you'll get better with practice and as you build a data collection.

See also Software Estimation: Demystifying the Black Art, by Steve McConnell. This book is an excellent resource.

"Software Estimation: Demystifying the Black Art"

Time Tracking Input Mechanisms?

2009-01-19T13:00:00-05:00

Do time tracking tools have a low uptake because it's tedious to enter your activity data?

What do you use to enter time / activity data into your tracking tool?

If you don't use time tracking, is there a specific reason or is it just not something you've considered?

What if you could enter activity data through twitter or similar services?

Lastly, how much does the ability to (selectively?) share your activity data with others on related projects affect your choice of method?

Thanks for your response!

Simpler C Development Environment

2009-01-19T10:14:00-05:00

If you're on Windows and you had problems installing Cygwin, there is another package that includes GCC called "MinGW" (minimalist GNU for Windows). Download and run the MinGW installer.

Go through the installer and keep clicking the defaults. It will then download and install several other packages. When the install finishes, do Start Menu -> Run... -> "cmd", then:

C:\\\> md tutorial
C:\\\> cd tutorial
C:\\tutorial\> set path=C:\\MinGW\\bin;%path%
C:\\tutorial\> gcc hello.c

And follow the rest of the directions from the first tutorial post, making the appropriate path substitutions and using \ instead of /.

If you have problems downloading through the automatic installer, try following the MinGW "Manual Download" directions. (Just for the "minimum requirements".)

The Best Development Tools

2009-01-14T09:25:00-05:00

There are two kinds of great software development tools:

Those that do one thing exceptionally well.
Those that allow you to communicate better with your coworkers.

In some rare cases, you may find that the same tool lives in both categories.

Without an insane focus on exactly one mission, tools get watered down and cumbersome to use.

Quick example: version control tools. When evaluating a tool, ask what else the tool is capable of. If it can also handle bug tracking, feature requests, mailing list management, continuous integration, and monitoring the coffee pot, then back away slowly. Seriously. You want a version control system that is tightly focused on its core mission and nothing else.

Yes, you need those other capabilities, but you're better off getting them from a different tool and then integrating the two. I know, I know, it sounds like such a great idea to have everything under one roof, one vendor to contact, etc. The reality is that you end up with a comprehensive, tightly integrated set of weak tools.

Getting Started in C

2009-01-09T12:14:00-05:00

Some may argue that C is an outdated language to learn in 2009. I disagree. Learning C requires some degree of understanding how computers really work. And if you can understand pointers, you're ten steps ahead of the game.

This is the first in a series of articles designed to help the reader learn the C language and understand how to use it to solve practical problems. The tutorial will be based on gcc in a Cygwin or Linux environment.

If you are already running Linux or have Cygwin and GCC installed, you're halfway done with the first lesson.

Lesson 1

Part 1, Get GCC

If you're using linux, open a terminal window (bash prompt). Type "gcc --version". If you get back something like:

$ gcc --version
gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Then you're all set. If you don't have gcc installed, use your distribution's package installation tool to install it. (For example, Debian-based systems will use sudo apt-get install gcc.)

If you're on Windows, install Cygwin. When the selection box pops up to choose packages, expand the "Devel" category and select gcc:

There's a video tutorial available if you want more detail, or a text tutorial if you'd rather read about it.

Part 2, "Hello World"

Now you're ready to write your first program. If you're on Windows, I strongly recommend using a decent text editor instead of Windows Notepad or WordPad. If you have no editor on your system, try Notepad++. I haven't used it much, but it seems to work well enough. If you're on Linux, you have a choice of whatever editors are installed on your system: vi, emacs, pico, etc.

Now, create a directory to store your source files. On Windows, place this directory under your cygwin installation directory. I'm going to assume you're using c:\cygwin\tutorials\, so adjust accordingly if you choose a different path.

Finally, we can write a program to print the string "Hello World" to the output when you run it. Open your text editor and *type in* the following program. (Do not cut and paste. The act of retyping the code presented in this tutorial will help you learn better.) #include int main(int argc, char** argv) { printf("Hello World\n"); return 0; } Now, get to a bash prompt. (On cygwin, this is the cygwin icon that should be on your desktop from part 1 above. On linux, this is a terminal window or whatever your system happens to call it.) Change to the directory where your files are stored. bash$ cd /tutorial On cygwin, this will put you in the c:\\cygwin\\tutorial directory. Now, compile the file. bash$ gcc hello.c GCC does not give output for a successful compile, it just produces the executable. If you do a directory listing ("ls" at the bash prompt) you will see a file called a.out (linux) or a.exe (cygwin). For historical reasons, this is what the compiler produces by default. You can run this and see the output. Type "./a.exe" or (./a.out) at the bash prompt and you should get the greeting.

We can change the output file by telling the compiler what we want it to name the file. bash$ gcc -o hello hello.c The -o flag (think of "output") tells the compiler to produce a file called hello.exe (cygwin) or hello (linux). Try this, and run "./hello.exe". ### Part 3, Breakdown Here's a rundown of what the pieces of this program mean.

#include <stdio.h>

This tells the compiler to pull the text of the file stdio.h into the program. This is called a "header" file, because it is typically included at the top (head) of your program. This header file contains the standard input/output definitions and declarations, including the declaration of the function "printf" which is used below.

int main(int argc, char** argv)

This is the start of a function definition. "main" is the name of the function that the operating system will call when you run the program. Main is always declared this way: it accepts two "arguments" called argc and argv, and it returns an integer (int). We'll come back to the argc and argv in another lesson.

{

A left brace is used to indicate the start of a block. The line that describes the type and arguments of the function is followed by a block that contains the programming statements that actually do something useful.

printf("Hello World\n");

This calls the printf function, which sends text to the output device (in this case, your terminal window). A function call is built by following the function name with parentheses and including arguments inside the parentheses. The printf function is passed an argument: the string "Hello World\n".

In C, strings are surrounded by quotation marks. The sequence \n means "newline". Exercise: modify the program to remove the \n, compile it, and see what happens.

return 0;

This ends the function, returning control to the function that called it (in this case, back to the operating system). It gives the value 0 to the calling function. When we return 0 to the operating system, this signifies a normal exit. Returning 1 or other values generally signifies that an error occurred.

}

A right brace is used to mark the end of the function.

Exercise: Experiment with this. Change the string. Add another call to printf to display a different greeting.

Next lesson: the debugger.

Version Control Habits of Effective Developers

2009-01-07T09:55:00-05:00

(Apologies to Stephen R Covey for the title...)

Through some odd coincidence these posts crossed my path recently:

Check In Early, Check In Often [comments here are especially good]
Best Practices for Version Control
5 SVN Best Practices
Continuous Integration is an Attitude, Not a Tool

What I got to thinking about is that version control is one of the foundation tools in your development process. It enables many other processes and practices. The posts above make great points, but I'd like to put them in the context of the overall process. (I've also given some thought to the conflicting advice given, again in the context of the forest instead of the trees.)

Use version control! If you're not doing this, you should get out of the profession now.
Check in often. This is frequently given advice, but I feel it's often given out of context. [Jeff Atwood][Check In Early, Check In Often] includes a quote from AccuRev founder Damon Poole: My rule of thumb is "check-in early and often", but with the caveat that you have access to private versioning. If a check-in is immediately visible to other users, then you run the risk of introducing immature changes and/or breaking the build. (My emphasis.) "Private versioning" means a private branch: you can check in without affecting anyone else (see #8). But even with a private branch, consider Salvatore Iovene's advice: "SVN is not a backup tool". He makes the point that you shouldn't check in before you go home just because it's the end of the day. Instead check in when you have a logical unit of work (e.g. everything compiles, most of the tests pass but you have to make some experimental changes to finish the feature or perform debugging, etc.). The danger is that if you have to revert because you've made some disastrous changes, you won't have a sane state to revert to. Final point: Salvatore also reminds us that some tools don't commit to a central repository by default (git, darcs, etc), so checking in doesn't replace a good backup.
Put everything in version control. This is a key enabler for continuous integration (CI). James Shore has several great posts on CI. I think he under-emphasizes the need to check in everything necessary to build. First, it should be easy for a new hire to pull down a source tree and immediately get a good build. It makes life easier for them and for you. Second, your CI server should be able to do the same thing. If someone upgrades one of the tools locally and the build depends on this then the integration build should break. The tool upgrade must be checked in so that everyone picks up the change.
Don't break the build. [When I say "build" I mean "build and tests". It's unfortunate naming but I think this is the way the term is used by a lot of people.] This is a promise that the team has to make to each other: "I will not break the build." What it means is that every team member agrees not to check in code to the trunk that does not build and pass all tests. If you have jerks on your team that constantly break the build, ask them to play by the rules, and if they can't then get rid of them because they're dragging you down.
Check in bite sized chunks. Too big and you've probably been working too long: did you go dark? Large checkins are also harder to review well, so quality will suffer. If a given feature is large, create a branch and continue to check in bite sized chunks (see #6).
Branch, but only when needed. If you don't have bite sized chunks, or if more than one person needs to work on a feature that can't go into the trunk, create a branch. When you have a branch, it is critical to update (or merge, or pull, whatever terminology you prefer) into the branch from the trunk on a regular basis. Integration will be a nightmare if you don't. When the code on the branch is mature enough, merge (or push) it back to the trunk. (Tip: it doesn't need to be perfect, it just shouldn't introduce regressions into the trunk.) If more non-bite-sized work is needed on the feature, create a new branch and repeat the process.
Check in when you're ready. But not sooner or later. This should be obvious, but some people don't seem to get it. If you check in before you're ready, you break the build (and your promise to your team members). This can stop your entire team dead in their tracks! If you wait too long, you've lost the opportunity to get rapid feedback on your work. This practice enables continuous integration. It relies on having work broken down into bite sized chunks (see #5).
Use private versioning. Private versioning means that each developer has his own branch (or even multiple branches). With a VC tool like AccuRev, this is automatic: every tree has an associated private stream (branch) on the server. With distributed VC tools like git, this is also automatic: everything is private until you push/pull to the central repository or another developer. It can be simulated with other tools but requires a bit more effort. This allows individual developers to maintain several versions of their work before they are ready to share with the rest of the team. It also allows a developer to keep a sane version of code before merging in changes from the trunk. (If the merge becomes a disaster, he can discard the merge results and go back to the sane version without much effort.)
Update/merge/pull regularly. This also seems obvious, but again not everybody gets it. I mentioned it above as part of branching, and it's especially important if you're using private branches. You must merge from the trunk to your private branch/workspace/sandbox regularly. If you have private versioning, this is low risk because it's easy to revert a bad merge. (These two practices create a self-reinforcing virtuous cycle!)
Train your users. I've worked in shops where it seems like only half of the team understands anything beyond "checkout, update, checkin". Everyone must know how to perform daily duties like checkout, checkin, update and merge from trunk, merge from private branch to trunk, how to navigate history, perform diffs, etc. Everyone should know how to perform less-frequently-needed duties like odd sorts of merges - cherry picking, etc, though this is maybe less important.
Use a common project structure and naming convention. Thanks to Anders Sandvig for this bit of wisdom. I wouldn't have thought to include it in VC best practices but it makes sense to list it here. A common project structure makes it easier to add every project into the CI server. For example, I've used the CI server to enforce the rule that every project must provide a $BUILDROOT/build script that runs the entire build. (The server also enforced placing tests in $BUILDROOT/tests, list of images to be published in $BUILDROOT/MANIFEST, etc.) This also makes it easier to move staff from one project to another.
Enable checkin notifications. This is typically seen as an email from a post-commit-hook (or whatever your tool calls it). The checkin notification can also launch CI build. Ideally, the notification will tie each checkin back to whatever issue/defect/task/project tracker you're using.
Write good checkin comments. This makes notifications worthwhile. Remember that VC is a communication tool as much as a time machine! I've had several experiences where reading someone else's checkin comment has spurred a discussion that either causes them to rework their bug fix or it has caused me to change course on something I've got in progress.
Perform atomic checkins. - When it's time to check in, don't do ten different checkins for the ten different files you modified! They all belong together as a logical unit. This makes merging easier. It prevents the CI server from running multiple build and test cycles needlessly (and from reporting spurious breakages because it picked up half a change). This makes history easier to understand when others look at comments. This is part of training, and is reinforced if you expect all developers to merge regularly. (On the theory that merging is easier when checkins are atomic.)
[Advanced practice.] Use a holding area. - Instead of checking in to trunk, developers check in to a holding branch (call it "pre-integration" or something similar). You perform code review and run integration tests on that branch. When review has approved the changes and the tests pass, promote (merge) the change from the holding area to trunk. The rest of the team pulls (merges) from the trunk, which is guaranteed safe. I've also seen the use of a two-level holding area where the first level is for review and the second level is integration test. This is the default if you use private branches (#8) and perform code reviews out of the private branch before integrating (#16).
Perform reviews on checked-in code. This is so much better than manually gathering changes and emailing patches around. (Though tools like git have optimizations specifically for eamiling patches around.) This is easy if you're using private branches since you can request a review before integrating. Some code review tools enable this.

Continuous Integration Tool Roundup

2009-01-05T13:31:00-05:00

This is a round-up of continuous integration tools, in no particular order.

CruiseControl: Open source, Java-based. Focused on build.
Cruise: Commercial. Build/test/deploy/release. Works for most platforms and languages. Free for small and open source teams.
CruiseControl.NETOpen source rewrite of the original CruiseControl for Java.
Hudson: Open source, Java-based, runs as servlet. Supports build and test. Other processes supported via plugin architecture.
ElectricCommander: Commercial. Does build, test, deploy.
AnthillPro: Commercial. Does build, test, deploy, release.
Continuum: Open source from Apache. Focused on build.
Gump: Open source from Apache. Focuses on building and testing public open source Java projects. May not be suitable for private installations.
Bamboo: Commercial. Build and test, with other processes provided by plugin.
Automated Build Studio: Commercial. Build, test and release.
BuildBot: Open source. Build and test.
CABIE: Open source. Focus on build.
Cascade: Commercial, free for personal use. Build and test; can be done before comitting changes.
Draco.NET: Open source. Focus on .NET (Windows only?) builds.
TeamCity: Commercial with free professional edition. Java and .NET only. Build, test, inspection (code review).
Team Foundation Server: Commercial from Microsoft. Team Build is the continuous integration component, tightly integrated with the rest of the system.
Tinderbox: Open source from Mozilla. Focuses on cross platform build and test.
Parabuild: Commercial, free for Open Source projects. Does build, test, deploy.
TapTinder: Open source. Build and test, with emphasis on test cases.
Pulse: Commercial, free for small teams and open source.
Homebrew: The pile of scripts and cron jobs you might be using to manage a daily/nightly/continuous build.

If you're in the Homebrew camp, please leave a comment: I'm curious about what the tools above are missing that force you to DIY. (I've been in this category in the past, for various reasons.)

I'd also appreciate any comments as to the relative merits of the systems mentioned above. I've only used a couple of them.

Lastly, if you use (or sell) some tool that is not listed here, please leave a comment and I'll update the list. Thanks!

Contact Me

2009-01-04T18:50:00-05:00

There used to be a form here to contact me, but it is no longer here.

You can send email to the obvious username at this domain.

Migrated to WordPress

2009-01-03T20:43:00-05:00

Just switched from Blogger to WordPress. Several links are broken (I will fix these soon), and it looks like subscribers are probably going to see a bunch of duplicate posts. Sorry about that...

Removing Defects From Django Apps

2009-01-02T07:29:00-05:00

I'm working on a set of Django apps right now. Here are some of the things I'm doing to filter out defects as I work:

Design Review

I do a handwritten (paper) design. I like to design offline — I find that this forces me to do two things. First, to slow down and get it right. Second, to stay at about the right level of abstraction instead of drilling down into too much detail. When I'm done the paper design, I review it against a checklist that I build from mistakes found in downstream phases. My design reviews catch about 70% of the mistakes I make while designing. I'd like that to be higher, but it is definitely worth taking the time to do it. This isn't specific to Django or Python, I do this for all projects.

Written Test Plan

After design review is done, I write (on paper, informally) a unit test plan based on the design. This is only possible when the design is at a low enough level that you can know what the general flow of control looks like through each method. I aim for full branch coverage in my unit tests. This doesn't mean it will be "fully tested", but at least I'll have exercised most of the code. I usually find (and fix) a couple of defects while writing the test plan. This is also something I do for all projects.

Code Review

After I'm done coding, and before I run any of the code, I print it out and review it offline. The following bash alias is helpful (prints 2-up):

# usage: py2pdf OUTPUTFILE INPUTFILE(s)
py2pdf ()
{
    out=$1;
    if [ -f $1 ]; then
        echo $1 exists, aborting;
    else
        shift;
        echo $*;
        a2ps -A fill -E -2 -C -o - $* | ps2pdf - > $out;
    fi
}

It's controversial, but I like the advice from Watts Humphrey in the PSP: review code before compiling (or in this case, before running any unit tests since there isn't really a compile phase). It forces me to be much more thorough looking for defects since I know that python hasn't filtered anything out yet.

To review the code, I use my checklist which is built in the same way as the design review checklist. I also use the test plan, walking through the code with each test case to see if there are any defects that testing will expose. It's better to fix these now instead of waiting for the unit test to find them. Two reasons: First, it's faster because you can see what the problem is. Second, you may notice something that the unit test won't catch for whatever reason — maybe you don't end up including the right data pattern to catch the defect. Code review is a generic defect removal practice.

Unit Test - XHTML Validation

Again, this seems to be controversial though I don't really understand why. Some people don't think valid XHTML is worth much. I validate because I want to make sure that my generated content isn't horribly broken, and — just like compiler warnings — you either validate cleanly or you don't validate at all. As soon as you start ignoring "a couple of silly errors" everything goes downhill.

I wrote an XHTML validator middleware for Django but I haven't yet used it on this project. Why not? Well, I was trying to use Dojo as a javascript library, and it's not possible (or at least not easy) to write valid XHTML while using Dojo widgets. So I had to write some special code in my test harness to strip out all of the Dojo garbage and then validate the result.

I'm using jQuery now, I'm much happier, and I suspect I'll be able to validate using the middleware on my next iteration. (Which involves ripping out the Dojo false starts.)

Validation is generic for any web project. The middleware is specific to Django.

Unit Test - Test Plan

I implement the written test plan using PyUnit, the Django enhancements based on PyUnit, and Selenium for browser-based testing.

My test plan usually generates a lot of test cases (a recent app had over 100 test cases for about 800 lines of Python, 700 of Javascript, and 200 XHTML). So I like to keep them in separate files in a "tests" directory in each app. One trick that I use is to implement a suite() function in my APP/tests/__init__.py. The suite function globs "APP_tc_*.py" and sorts them by time stamp, most recent first. This way if a test is failing I can make sure that it runs first when I fix the problem and rerun the suite. The other part of the trick is to add an optional argument to the globbing/sorting function that limits the number of test cases returned. So when I need to rerun a test a few times to fix a tricky problem, I don't have to wait for the entire suite to run to get the results of the test-under-development. This is all highly specific to Django development.

Unit Test - Pylint

This is still experimental for me. Pylint generates a lot of noise. Most of what it reports is legitimate, but about 90% are minor (i.e. not-user-impacting) defects. I've had to spend a fair amount of time tuning the report parameters to stifle the noise. The most time consuming part so far has been filtering out results from the test code. I'd like to figure out a way to make pylint skip the test files but there doesn't seem to be a setting for this. On the bright side, it nagged me into making some of my test code more maintainable. Obviously pylint is specific to python, I use other lint-like tools for other projects when the signal-to-noise ratio is high enough.

Unit Test - jslint

I'm a javascript novice, so jslint has been helpful in ferreting out problems that I would not have caught through testing. As I let the feedback from the tool and testing form better design and coding habits/patterns, the signal-to-noise ratio may drop so that this is less effective.

Productivity: It Comes from Software Design Rather than Software Tools

2008-12-31T11:51:00-05:00

I just read Scott Bellware's [Productivity: It Comes from Software Design Rather than Software Tools][].

These bullet points — the core of his argument — are excellent (quoted here): > - Design quality is the most important factor influencing > productivity in software development > - The things that obstruct quality degrade productivity > - The reductions in productivity over time that are typical to > tool-driven software development are greater than what can be > solved by tools > - The application of tools to these problems exacerbates the quality > problem, creating a vicious cycle that accelerates exponentially > - Quality software design is the product of principles and > practices, not tools > - The typical degradation in a software's quality over time isn't > due to the nature of software, it's due to the nature of the > approaches we choose to develop and operate software Full disclosure here: I am working on a set of software tools, and a big part of my consulting for software teams is helping them install tools to support their improvement initiatives. I agree with Scott that good design is the root of all that's virtuous in software. I guess I also agree that tools are often a distraction, though I think he's drawing a direct line where the root causes go deeper. Maybe it's my embedded background, and the fact that I've worked on teams that were woefully under-equipped in terms of tools. (Or execs were sold a bill of goods and we got stuck with the resulting boat anchors and paperweights.) I just don't see an over-emphasis on tools, maybe this happens in IT more. What I do see are many opportunities for improvement. - Designs can be better communicated. If designs can be communicated better amongst the team, they can be reviewed better, which will kill bad designs before they get coded — and the feedback will help developers improve their future designs. A tool for formatting and transmission could help here. But such a tool would only be helpful to a team that already has processes in place for creating and reviewing the design. - Requirements can be managed better. Practices like Scrum seem helpful, but there are still challenges. Again, a tool can help here but only when the team has an understanding of how the tool fits into an existing process. - Existing processes and tools are overly slanted towards managing code. [By the time any requirement or design has been reduced to code, it's way too late to fix it.][] Start working on improving quality earlier in the cycle, the effort will pay off in a big way. Putting software tools in the hands of unskilled developers is like letting me go after wood with power tools: it's an expensive way to make a lot of sawdust. Better to build skills with manual processes and then introduce tools when manual chores become tedious. [Productivity: It Comes from Software Design Rather than Software Tools]: http://blog.scottbellware.com/2008/12/productivity-it-comes-from-software.html [By the time any requirement or design has been reduced to code, it's way too late to fix it.]: http://blog.bstpierre.org/five-reasons-to-slow-down

Code Review Tools

2008-12-31T11:07:00-05:00

Yesterday I posted twenty reasons to do code reviews, and I promised a list of code review tools. Here they are, in no particular order. I have not used all of them, so I can't comment on their relative merits. If there are some I missed, please leave a comment and I'll update this list.

Codestriker Open source, web based, written in perl I think.
Code Collaborator Commercial.
Review Board Open source, web based, appears to be python/Django.
Rietveld Open source from Google. Guido van Rossum's 2nd pass at a code review tool. You get three guesses on the implementation language...
Crucible Commercial from Atlassian.

Bottom line: don't try to do this by hand, you'll be wasting a bunch of time packaging patches and tracking defects when a tool can do it automagically for you.

Twenty Reasons To Do Code Reviews

2008-12-30T11:06:00-05:00

Update [2008-12-31]: I posted the list of code review tools as promised below.

I tweeted this article on Five Reasons to Do Code Reviews from CIO.com last week,and realized that there are much more than the five reasons they give. So I came up with 20 more over the rest of the day. This is a collection of those tweets, with a little more detail given here.

Code review reason 1: Feedback comes back fast enough to give you whiplash. Since code review is done after coding and before integration or system tests, developers don't have to wait as long to get feedback on code quality. By providing specific, timely feedback, developers can adjust their coding practices to avoid common mistakes.

Code review reason 2: Find more bugs than testing alone. This is obvious: Testing finds bugs. Code review finds different bugs. When combined they find more bugs than you would otherwise remove before shipping to customers.

Code review reason 3: A million monkeys sometimes produce valid syntax, even if it doesn't make any sense. Compilers can't catch this.Sometimes you'll make a syntax error that produces valid syntax so the compiler can't catch it. The classic here is in C, = (assignment) versus == (boolean equality operator); in this case most compilers can issue a warning but there are other examples. A code review will catch this error quickly, but finding this during testing can be hard to track down.

Code review reason 4: Find and kill bugs where they live, instead of finding turds and having to track them back to the nest. When you get defect reports from a code review, you have in-hand the exact line number and problem description. When you get reports from system test or (horrors!) a customer, you get — at best — a reproducible set of steps to trigger the bug. It's then up to you to work backwards to the actual defect in the code, which is often time consuming, especially since the code may have been written weeks or months earlier. (See reason #1.)

Code review reason 5: Indoctrinate the n00bs. Code reviews are an effective training tool. First, new software engineers who participate in code reviews get to look at code produced by other engineers and thus get a feel for the coding standard and documentation style. Second, inexperienced engineers will get feedback on their code before they have a chance to infect the system with inferior code. (Because we all know that the code we wrote 10 years ago probably sucks — unless we were fortunate enough to have it reviewed by more senior engineers. If you haven't already been coding for several years, keep in mind that everything you're writing today probably sucks but you won't be able to acknowledge that for another few years; see reason #6.)

Code review reason 6: Uses developer egos to boost quality of code from the beginning. The theory here is that because developers know their code is going to be publicly scrutinized by a group of peers, they will take extra care in coding to avoid the embarrassment caused by having others find all kinds of bugs in their code.

Guest contributor @kyletolle: [Code review reason [7]][]: Explaining code makes sure they understand it well. Will fix the "Uhh, b/c I always do it that way" remarks. After you have to answer this question a few times during a review, you'll probably ask the question of yourself while coding. (See reason #6.)

Code review reason 8: Makes testing go faster. Heck, it makes the entire development cycle go faster. There are multiple root causes for this. First, since code review finds and fixes defects at the site of the defect (see reason #4), less time is spent fixing these defect. Second, fewer bugs are found by testers so they spend less time writing up issues and the engineering team spends less time tracking and managing issues. Third, testers don't have to re-run regression tests so many times because fewer release candidates will be needed.. Last, it is less likely that testers will be blocked waiting for fixes to bugs.

Code review reason 9: Fewer angry phone calls means happier tech support staff. Proof: Fewer bugs in shipping code (see reason #2). Customers not disrupted by bugs as often. They don't get angry as often. So when they call, it's with songs of praise. QED. (Ok, maybe that singing part isn't true, but you get the picture.) To prove this from another angle: if you're spending less time fixing bugs, you can spend more time implementing features that customers are requesting.

Code review reason 10: (Anti-reason) You'll pay more sales commissions because you'll be selling more product...wait, that's a good thing! When customers sing your praises because your product is so reliable, it's easier both to sell to new customers and to sell more product to your existing customers. Sure, your sales commissions might go up, but I've never really heard anyone complain about this too loudly.

Code review reason 11: There are tools that reduce the annoyance factor of old school review processes. After I tweeted this, @kyletolleasked for some recommendations. I will follow up this post with several tools that I've found. Stay tuned. (Look at my timeline starting here for now).

Code review reason 12: There's zero risk. Even not-super-effective reviews beat testing. If I spent a little time I could probably dig up some published research to back this up. But my personal data says my personal code reviews are about four times as effective as unit testing. So even if I perform a half-hearted review, I'm still going to be more effective than testing. (See reasons #2 and #4, also keep in mind reason #5 which doesn't have an immediate measurable effect.) There might be data out there that says code reviews are a waste of time. Flying reindeer might also exist.

Code review reason 13: (Anti-reason) Testers bored from not finding as many bugs... Oh yeah! They could focus on stress/perf tests. This would be a good problem to have...

Code review reason 14: Reviewing code is good practice for reviewing code. You'll get better. Your team's review effectiveness will improve over time. This means that the long term risk of failure is even lower than the short term risk (see reason #12 for why the risk of failure is zero or very close to zero).

Code review reason 15: It's a great way to build a really useful code review checklist. This is kind of stretching, a useful code review checklist doesn't really matter to the customer. (I should probably come up with a couple of bonus reasons to account for the "artificial fillers"... Hey, I tweeted this during the 3pm slump before my afternoon cup of tea.)

Code review reason 16: Out on a speculative limb: better staff retention because developers spend less time fixing annoying bugs.I'm not sure about anyone else, but I've stayed at a job that was otherwise not-so-great because of the quality of the team I was on and the people around me. Being part of a well-oiled machine feels good. It's motivating to go to work everyday knowing that you're going to produce a high quality product and your coworkers are there to back you up.

Code review reason 17: Reduces schedule risk because product quality is higher as it enters downstream phases. Your testers won't get blocked as much (see reason #8) and testing won't drag out. Since you get feedback relatively early in the schedule (see reason #1) you have a good idea of how long testing will take.

Code review reason 18: Feedback (especially from peers) at the source level means you will stop making the same mistakes over and over.(See reason #6.) This also stems from the fact that each engineer will have data in-hand about the types of defects he makes and he can come up with ways to prevent this defect from happening to begin with. And this is really when the magic occurs: when you can start thinking about preventing entire classes of defects.

Code review reason 19: Improves the quality of documentation / comments as well as executable code. (See reason #7.) Developers will include better comments to avoid having to answer too many questions from reviewers. If reviewers are looking for inconsistencies between comments and code, this is extra motivation to make sure that code documentation is up to date. Lastly, I suspect that overly complex code will become scarce since this will be scrutinized and questioned more than simple code passages.

Code review reason 20: Customers won't find as many bugs in their soup. This is maybe redundant to reason #9, but helping customers is why we exist, right? So I'll allow this one to stand and — like I said above — I'll work on a couple of bonus reasons to add to this list.

[Code review reason [7]]: http://twitter.com/kyletolle/status/1076701951

The cost of (not) testing software

2008-12-24T06:31:00-05:00

Great post on the cost of (not) testing software. The take-home lesson is "find defects early".

The main thing missing from the discussion is that there are better techniques for finding defects than testing. Like design and code reviews, and especially more attention to requirements. Catch defects as early as possible and reduce costs even further.

Project Team Blog: Our Projects Are Always Late

2008-12-18T07:09:00-05:00

Saw this post over at the Project Team Blog Our Projects Are Always Late. Newshirt asks (I'm paraphrasing) "Why would this person not use a time tracking tool?"

Two answers:

Because it feels like an extra step. If you want to change a behavior like time tracking, you have to make it automatic. Developers and other project members are already too busy, you can't just ask them to add another step to their daily workflow unless the benefits are immediate and obvious. The benefits of time tracking are not immediate and obvious -- it takes some time (granted a small amount of time) to reap the benefits of time tracking.
Second, they perceive a risk. Even if the software is free (they are providing a 30 day free download), there's the risk that you spend an hour or two (or more) downloading and fiddling with the software only to discover that it won't work in your environment for one reason or another.

To use concrete examples from my recent experience:

I'm developing in Python. I've used Pylint in the past and it has been helpful in pointing out bugs before I start testing. I want to use Pylint on my current project, but I keep skipping it because it feels like an extra step and I'm trying to get work done, not fiddle with extra tools. The solution is for me to integrate this tool into my workflow so that it is automatic and not an extra step. This is what I've done for the current iteration and I expect that it will stick.
I've recently been using more social media sites like twitter and I read about a toolbar for Firefox from Minggl. Sounded interesting and after I saw a few mentions of it I decided to give it a whirl. There should be a warning: "roadblocks ahead".
1. They make you register to download.
2. The registration form does not allow periods or spaces in your last name, so it took a couple of tries to register. (This smacks of ethnocentrism! I am being oppressed because I have a period and a space in my last name!)
3. It doesn't work on linux... bummer.
Contrast this with the experience of signing up for something like twitter or Mahalo Answers. You sign up (albeit with the gotcha of only allowing alphabetics in the name) and you start using it immediately.

Five Reasons to Slow Down

2008-12-15T11:22:00-05:00

Here are five reasons you should wait before moving on to the next phase of your software development process:

Get the requirements right. It's so often repeated that it's almost a cliché to say that requirements errors will cost 10x or more to fix during coding or testing. But it's true, so I can risk repeating it here. Spend a little more time getting the requirements right, and you'll get every second back, with interest during the testing phase. The risk here is overanalysis, but if you tend to be the impatient type -- jumping into design or coding with very thin requirements -- then you're unlikely to suffer paralysis.
Get the design right. It doesn't have to be an elaborate 500 page document. One of the best designs for a complex system that I've ever seen was a state machine described in a two-page document with a big matrix on the first page. Remember, a written design is a communication tool. Just like the rest of your software, it can be the simplest thing that will work. Marker on a whiteboard works, if your team decides that is an effective way to communicate. An extra hour spent preparing and reviewing the design will save at least two hours in testing. Really. I've seen too many designs that were rushed (and clearly broken upon review) because of an unexplainable urgent desire to get a document into review so that coding can begin. The state machine I just mentioned went through probably 20 man-hours of reviews in maybe three passes, and we must have found at least 50 defects in it. Those defects took a few seconds each to fix (some may have taken a few minutes), but if there had been code and tests based on this broken design it would have taken hours to fix some of them.
Get the code right. I used an Extreme Programming mantra above ("the simplest thing that will work"), but I think XP overall often leads to bad practices -- primarily an overemphasis on testing. I agree that automated regression tests are a valuable tool. However, it is important to recognize that testing has limits. It takes an immense effort to achieve full branch coverage, and even this can't be considered "complete" testing. Testing must be used in conjunction with other tools: code review and static analysis are two such tools for removing defects from code. And before you're at the point where code is ready for review or even the compiler, take the time to consult a reference when you're unsure how to use an API. I don't know how many bugs I've fixed (including some of my own!) that were a few lines down from a /*TBD: how is this supposed to work?*/ comment. Instead of putting in that comment, take the time to figure out how it is supposed to work.
Get the tests right. Ever waste time on a bogus bug report? Testers telling you a feature is broken when they're just doing the wrong thing? It's not their fault if you failed to communicate how the feature is supposed to work. In the same way your test team should be insisting on some written communication from you about the feature set, you should insist on some written communication from them about what they're going to test. Again, this "test plan" doesn't have to conform to an ISO/IEEE standard that requires it be 1200 pages long and six inches thick. Like with the design, a whiteboard works just fine. Whatever communication method you use, if it helps head off problems then it is working. If you're finding too many problems late in the game, then maybe you need to have something a little more formal -- try posting short documents to an internal wiki.
Are you keeping score? A little bit of measurement can do great things for you. I track time, lines of code, and defects. I was reviewing one of my designs this morning and this data helped me decide: (a) how many defects I should expect to find in my design and (b) how long it should take me to find those defects. All because I have historical data that tells me how many defects I find per hour of design review, and how many defects I make per hour while designing. I also know how much testing time this is likely to save... so I don't feel any particular rush to start coding from a buggy design. I'd rather get the design right first.

As I mentioned above, the risk to getting into the habit of waiting is that you spend too much time analyzing the current phase and don't just get on with it. But if you start from good requirements, it should be obvious when you're done with design and coding. And if you're keeping score, you'll know how much time to devote to reviews to keep them efficient.

Amnesia - Joel on Software

2008-12-10T05:49:00-05:00

Joel Spolsky also thinks that using timelog data for performance tracking is a bad idea.

Jason Cohen on QA vs.QC

2008-12-09T07:08:00-05:00

Jason Cohen of Smart Bear Software has a brilliant blog. His post on QA vs. QC is something I've talked about in the past. I love his example of the Pringles manufacturing line discarding off-color chips -- it's a concrete, understandable way to define the difference between QA and QC.

Some of my best project experiences have been those that have involved the testers starting at the requirements phase. These guys have a different attitude and have asked some great questions that prevented us from going down dead-ends.

Great testers find and isolate really hairy bugs. Truly great testers not only find these bugs, but will often find a pattern of bugs and can, for example, point to the portion of the spec that the code failed to implement. They take a bigger picture view of the product and its environment and how to improve the quality by both preventing defects and figuring out how to detect defects during the test phase.

As Jason says, we should let these guys have a little more influence on the development process. For example, they should be insisting on design and code reviews, even if they don't participate. Because when the development team is in solely in charge of the development process, these steps sometimes get skipped -- because the job is a rush, or because the feature is small and doesn't need to be reviewed. (Even though rush jobs are most in need of reviews, and small features always seem to take longer because they receive less care and attention.)

Eight Reasons to Use a Time Tracking Tool

2008-12-08T08:41:00-05:00

I just finished the time tracking application for Bale. This is a tool that is key to many other practices, but is very underutilized. (I wonder if this is because so many developers are worried about Big Brother type intrusions into their working habits? See below for why Big Brother is such a bad idea. If your employer is in this camp)

Here are eight reasons why you should use a time tracking tool:

Gather reliable data for future estimates. You can't make accurate estimates without accurate historical data. If you try to create estimates using a simple count of days spent working on a project, you'll be way off because so much time is lost to non-project activities (like supporting the last project you were on). A time tracking tool keeps an accurate record of how many hours you actually spent working on the project.
Eliminate time wasters. When you start tracking time, you'll be surprised how little time you spend working on your current project. You can begin to pay attention to the low value activities that come up during the day. Awareness is the first step, then you can start to reduce or eliminate these time wasters so you can put more time into your bottom line.
Evaluate new practices. Without good time data you have no idea if the introduction of a new practice or process step is helping or hurting. For example, say you move to performing code reviews online instead of on paper. You may have a good idea of how much money this will save in terms of ink and paper, but you have no idea whether it is actually faster. (I'll soon have a post on why defect tracking is important too -- evaluating new practices is a key benefit there too.)
Find areas for improvement. When I first started tracking time I realized the vast amount of time I spent in unit testing -- even worse, sometimes in system testing. I made a conscious effort to slow down when doing design work and it paid off. Hasty designs end up requiring a lot of rework during test. Your situation may be different, but you won't know until you start tracking where in the lifecycle your time is consumed.
Improve reviews. An effective review finds more defects per hour than an ineffective review. You can't tell the difference between effective and ineffective reviews without having the time component. This is sort of a subcategory of finding areas for improvement. For example, I know that my personal design review is about 4x more effective than unit testing, and slightly more effective than my personal code review. In a team context, knowing your team's defect removal rate during inspections is an excellent way to justify the investment in review time to upper management. This is why a team-wide time tracking tool is important.
Evaluate new tools. Too often, the process of deciding whether a new tool is going to work is based on watching the salesman demonstrate the tool on the projector in a darkened room, and then some haggling by upper management. Sometimes you get a 30 day eval, but in terms of a test drive, this is really just a spin around the block to see if it "feels nice". It's much more useful to be able to know that a given tool actually increased productivity by 8% during the eval period. This also provides leverage during the aforementioned haggling when the vendor is claiming 77% productivity gains...
Know when to quit. Ever had your manager say, "Don't spend more than an hour on it."? Ever blown the afternoon after hearing such a request because you just couldn't put it down?
Improve visibility. You can generate better upwards-facing status reports with a good time log. Remind your manager that the reason you missed the Friday deadline is because the VP of Engineering retasked you for all of Tuesday and Wednesday on his pet project. She can then show your status report to the VP as a reminder. (And oh, by the way, she can also show it to the VP of Marketing and CTO/Founder who're screaming about the missed deadline. Yes, your organization is not the only dysfunctional group out there.)

For all the pointy haired bosses out there, here are a few reasons to not use time tracking to compare employees based on productivity.

It's easy to game the system. You can enter fake data. This will, of course, screw up estimates and make it impossible to use the data for any kind of overall improvements. Developers need every assurance that the data will not be used for evil or they will see to it that the data is worthless.
Productivity varies by task. I've seen examples, on the same project, by the same person (me), where different tasks have enormously different productivity. Some problems are simply harder to solve. Sometimes you have to modify an existing bit of code: sometimes the existing code is easy to work with, sometimes it is a horrible mess. Sometimes you don't even know what the problem is, and you have to do a bit of research just to figure out what needs to happen to get something to work.
People make varying contributions. I've known people who appear to work very slowly, producing very little every week. Sometimes ("The Slacker") they're not actually working, instead day trading or playing solitaire. Sometimes ("The Craftsman") they are doing very solid work; you'll never see a bug come out of system test for these guys. Some developers ("The Enabler") don't do lots of work of their own, but rather spend much of their time enabling others, or perhaps interacting with marketing, support, or testers; this is valuable time spent, they are keeping your team in the loop or solving customer problems. You can't tell the difference with a time tracking tool, the slacker will simply enter fake data, and the enabler's contributions don't show up on paper.

If you're trying to compare using time log data, then when lean times come you may end up firing the Enabler because the Slacker looks good. You'll still have the Craftsman, but because he depended on the Enabler for answering questions, keeping Customer Support off his back, and keeping him in the loop about What Marketing Wants This Week, his productivity will drop dramatically.

Visions of Tool Integration

2008-11-21T14:56:00-05:00

It's an amazing thing when your development tools work together.

In the bad old days, we thought it was cool when the editor could work in concert with source control. It blew my mind when we first had source control with integrated bug tracking. (Wow, you can see the changes that go along with each bug fix!) In some ways, we've come a long way, but I still see a lot of integration that is within reach but still doesn't exist.

Look at something like Trac, which is open source and widely used. It is described as a "Project management and bug/issue tracking system. Provides an interface to Subversion and an integrated wiki." If you've seen it in action, it is an effective combination of the tools in that description. However, it is missing a few tools that are key pieces in an effective development process:

Integration with code review tools like codestriker or CodeCollaborator from Smartbear.
Integration with test automation.
Integration with static analysis tools --- lint-like tools or (at the high end) Coverity.
Integration with runtime analysis tools like valgrind.

Imagine the following scenario with all of the above features:

Alice finishes coding and testing the fix to bug 1257 and checks in the fix, all from within her favorite editor. The source control backend notifies the build server and a new build is launched. When the build completes it automatically kicks off a regression test. Whoops! Alice reintroduced bug 1142 with this fix. She checks in a new patch and this time all tests pass. The backend automatically adds notes to bugs 1257 and 1142 both when the regression occurred and when it was fixed.

Bob was already selected as the reviewer for this fix within the defect tracker, so he gets an email once the regression tests have passed to let him know that the patch is ready for review. He opens his browser to see the changeset diffed against the previous version. Alice added a couple of new function calls, and he clicks through to the function definition because the source display includes clickable cross-references. Whoops! Alice forgot to include a check for a NULL return from one of the functions. Bob clicks the call site to add a note, and rejects the patch.

Alice gets a notification about the rejection and sees the note Bob left. She adds a new test case, sees a crash from the NULL return, and then checks in another fix. When all tests pass, Bob gets another request for review. This time it all looks good, he approves the patch and it is applied to the development trunk. The backend makes notes in the appropriate issue tickets along the way.

Bob's review approval triggered a merge to the trunk, so the backend system automatically kicks off a trunk build and regression test. It also launches a full reanalysis using a commercial static analysis tool, and re-runs the regression test using valgrind. Because this is all automated and human intervention is only needed when something goes wrong, developers are free to focus on fixing reported bugs and adding features.

We aren't quite there today, but the individual pieces do exist, and fortunately there's no rocket science involved in putting the functionality together.

Five Simple Ways to Kill More Bugs Today

2008-11-21T12:42:00-05:00

Here are some simple things you can do - today - to find and prevent bugs in your code. These take very little effort to implement and do not cost any money. (And typically don't require management approval.)

Turn on warnings.
Make warnings break the build.
Review your code.
Get a review buddy.
Track all of your defects.
- Date
- Where found (file, class, and/or function/method)
- How found (self, compile, peer, test, qa, field)
- "Type" - some brief description to define the type or category of defect