The slides were generated using showoff.

The NoSQL movement has brought us a wave of new data stores beyond the traditional relational databases. These data stores come with their own tradeoffs, but they provide some incredible benefits. At Braintree, we are moving in the direction of using Riak as our next generation data store. We love its focus on scalability and availability. Servers can fail without causing any downtime, and we can add more capacity by simply adding more servers to the cluster.

One great feature of relational databases, however, is the consistency in the shape of the data. You know if you have a people table, every row has the same columns. Some fields might be null, but there won’t be any surprises. Furthermore, if you want to rename or modify a column, it’s a simple operation. In the case of PostgreSQL and other databases, a rename is nearly instantaneous. We lose this ability with Riak and most NoSQL databases. We can easily add attributes (columns), but we cannot easily rename them or change the data within each document (row).

Since our apps are always evolving at Braintree, we needed a way for our data to keep up with our code. Our solution is something we’re calling lazy data migrations, and we’ve built it into our repository and model framework, curator. You can read more about curator on our blog at Untangle Domain and Persistence Logic with Curator.

The problem

Say we have a collection of people in Riak. This is analogous to a people table in a relational database. When we first built the app, we added fields for first_name and last_name:

person = Person.new(:first_name => "Joe", :last_name => "Smith")

Some time has passed, our app has data, and we now realize that names are a pain. What do we do with middle names? What about people with multiple first or last names? We want to just simplify the system and collect only a name. We no longer care about a separate first and last name. The problem is we have a ton of data in the old format. How do we handle that old records have a first_name and last_name, but going forward, we want just name?

In a relational database, we would simply write a database migration that looks like:

ALTER TABLE people ADD COLUMN name VARCHAR;
UPDATE people SET name = first_name || ' ' || last_name;
ALTER TABLE people DROP COLUMN first_name, DROP COLUMN last_name;

This migration might take a while to run, but once it’s done, we know that all data has been migrated. We can then change all of our code to only deal with name, knowing we no longer have first_name or last_name.

In a NoSQL database like Riak, we cannot simply change the schema. We have to come up with a different solution. Here are the steps we went through in trying to come up with the solution that made its way into curator:

Solution attempt 1: Scattered conditionals

The first solution is to make the Person class smart enough to handle both cases.

class Person
  attr_accessor :first_name, :last_name, :name
end

We can populate whatever fields we get back from the data store. Then, when we want to do something with the name, we have to use code like:

if person.name
  puts "Name is #{person.name}"
else
  puts "Name is #{person.first_name} #{person.last_name}"
end

The problem with this approach is that we have to use branching code like this whenever we want to use the name. It quickly gets messy.

Solution attempt 2: Gathered conditionals

The second solution is to move this logic to the place where we read the Person out of the data store:

attributes = fetch_from_riak
if attributes[:name]
  person = Person.new(:name => attributes[:name])
else
  person = Person.new(:name => "#{attributes[:first_name]} #{attributes[:last_name]}")
end

Now, we only have to do it once and we can change our Person class to only know about name.

This solution works well, but what happens a year down the road when we’ve made lots of data changes to many different models? We don’t want a bunch of conditionals all over our persistence code.

Our solution: Lazy data migrations

We pulled the idea from solution 2 into the idea of a migration (similar to ActiveRecord migrations). Migrations target a given collection at a given version. They look like this:

class ConsolidateName < Curator::Migration
  def migrate(attributes)
    first_name = attributes.delete(:first_name)
    last_name = attributes.delete(:last_name)
    attributes.merge(:name => "#{first_name} #{last_name}")
  end
end

This migration is stored in db/migrate/people/0001_consolidate_name.rb. We’ve also added the concept of a version to each Model. By default, models start at version 0. When they are read from the Repository, the attributes are run through any migrations that are a greater version (based on the version in the filename):

person = PersonRepository.find_by_key("person_id")
person.version #=> 1

Now, the migration logic is isolated from the rest of the application. The rest of the app can safely assume that all Person objects have only a name:

class Person
  current_version 1
  attr_accessor :name
end

We mark the Person class with current_version 1 to signify that new instances start at version 1, since they have a name attribute rather than first_name/last_name.

These migrations run when models are read, so they are lazy. Data will migrate as it’s used, and update when saved. This means that, unlike with relational databases, the website can be up and serving requests while the data is migrated.

If you want to force the data to migrate (and not wait for all data to be used), you can simply find models who haven’t been migrated and save them. The version attribute is indexed by default:

PersonRepository.find_by_version(0).each do |person|
  PersonRepository.save(person)
end

Testing

Unlike ActiveRecord migrations, curator migrations have no side effects. They simply accept a hash and return a new hash. This makes them easy to call from a unit test:

require 'spec_helper'
require 'db/migrate/people/0001_consolidate_name'
 
describe ConsolidateName do
  describe "migrate" do
    it "concatenates first_name and last_name" do
      attributes = {:first_name => "Joe", :last_name => "Smith"}
      ConsolidateName.new(1).migrate(attributes)[:name].should == "Joe Smith"
    end
  end
end

Limitations

Curator migrations are lazy, so at any given time you might have documents with different versions in the data store. This is not normally a problem since the migrations will run as soon as the objects are read. However, if you add a migration that changes an indexed field, you cannot rely on that index to return all of the correct values until you migrate them all. In this case, you might want to force migration by reading and saving all of the documents.

Next Steps

You can see these migrations in action in the curator_rails_example.

Let us know what you think about lazy data migrations in curator. Feel free to open issues on GitHub, submit pull requests, and help us make it better.

The problem

Ruby on Rails is a great web framework, and we use it extensively at Braintree to build web apps. One criticism of Rails, however, is that it encourages tight coupling of domain and persistence layers. The convention in Ruby on Rails is for domain objects to extend ActiveRecord::Base and directly gain persistence. This makes building small apps easy, but as applications grow, the tight coupling starts to make things more difficult.

Domain objects in more complicated systems no longer have a simple mapping into a database row. For example, money might be represented as cents and currency in the database, but as a Money object in the domain layer. Now, the domain objects have to handle both the money logic and the money persistence. Even if the code is abstracted away by a gem, it still lives in the domain object. Take a look at the methods on a brand new model in Rails:

>> class Bar < ActiveRecord::Base
>> end
 
>> (Bar.methods.sort - Object.methods).count
=> 391

These methods are in addition to anything you add to the model.

The solution

There’s nothing in Rails that says you have to tie your domain objects to persistence. It’s merely a convention. At Braintree, we’re using a new convention for new projects. Our domain objects do not contain any persistence logic. Instead, we’ve introduced a repository layer which is in charge of the persistence of these domain objects. This pattern is well-documented in Domain Driven Design.

Introducing a Repository layer into our applications has a number of benefits. The main benefit is that it separates out different kinds of logic. Domain logic goes into the domain objects, and can be tested in isolation without any persistence. Persistence logic goes into the repository objects, which only deal with saving and retrieving objects.

The second big benefit is that abstracting persistence logic into a repository allows us to swap persistence back-ends without much trouble. The most notable case is that we can swap in an in-memory data store for testing (or for most tests), but use Riak for the real application. Since the repository interacts with the back-end data store, the application code is the same regardless of back-end. It also allows us to use different back-ends for different kinds of data but still have a consistent pattern around persistence.

We’ve extracted this model and repository framework from our current applications and released it as curator. Curator currently supports Riak as the persistence back-end, but more back-ends are coming soon.

Domain objects include Curator::Model, which just gives helper methods like an initialize that sets instance variables. It also adds some helper methods for Rails.

class Note
  include Curator::Model
  attr_accessor :id, :title, :description, :user_id
end
 
note = Note.new(:title => "My Note", :description => "My description")
puts note.description

These model classes don’t have a lot of extra stuff:

>> class Bar
>>   include Curator::Model
>> end
 
>> Bar.methods.sort - Object.methods
=> [:_to_partial_path, :current_version, :model_name, :version]

Repositories include the Curator::Repository module:

class NoteRepository
  include Curator::Repository
  indexed_fields :user_id
end

Repositories have save, find_by_id, and find_by methods for indexed fields:

note = Note.new(:user_id => "my_user")
NoteRepository.save(note)
 
NoteRepository.find_by_id(note.id)
NoteRepository.find_by_user_id("my_user")

As persistence gets more complicated, repositories can implement their own serialize and deserialize methods to handle any case. For example, suppose our note object contains a PDF:

class NoteRepository
  include Curator::Repository
 
  def self.serialize(note)
    attributes = super(note)
    attributes[:pdf] = Base64.encode64(note.pdf) if note.pdf
    attributes
  end
 
  def self.deserialize(attributes)
    note = super(attributes)
    note.pdf = Base64.decode64(attributes[:pdf]) if attributes[:pdf]
    note
  end
end

As you can see, all persistence logic around PDFs lives in the Repository. The Note class does not care how PDFs are stored.

Curator in action

If you want to see a simple, fully functional Rails application using curator, check out curator_rails_example. I will detail the relevant bits below.

Thanks to Rails 3.x, you can include ActiveModel::Validations in any class to get validations. So if you want your note class to validate, it would look like:

 class Note
   include Curator::Model
   include ActiveModel::Validations
   attr_accessor :id, :title, :description
   validates :title, :presence => true
 end

You can also build forms from these objects. new.html.erb would look like:

<%= form_for @note do |f| %>                                  
  <%= f.error_messages %>                                     
  <dl>                                                        
    <dt><%= f.label :title %></dt>                            
    <dd><%= f.text_field :title %></dd>                       
    <dt><%= f.label :description %></dt>                      
    <dd><%= f.text_area :description, :size => "60x12" %></dd>
  </dl>                                                       
  <%= f.submit "Create" %>                                    
<% end %>

And the controller looks like:

class NotesController < ActionController::Base
  def new
    @note = Note.new
  end
 
  def create
    @note = Note.new(request.POST[:note])
    if @note.valid?
      NoteRepository.save(@note)
      redirect_to notes_path
    else
      render :new
    end
  end
end

That’s it. Rails makes it really easy to build forms and validate against our models.
Next Steps

Please check out the curator code and let us know what you think. Like all software, curator is a work in progress, so feel free to open issues on Github, submit pull requests, and help us make it better.

I asked around for recommendations, and several of my coworkers recommended Zotac. After looking through their options, I decided on a Zotac ZBOX ID41 (here it is on Amazon).

Setting up Ubuntu

I decided to run Ubuntu 11.10 Oneiric Ocelot for the operating system. The Zotac ZBOX has no CD drive, so I downloaded the Ubuntu ISO from the website, and I used UNetbootin to make a bootable USB drive.

I plugged an HDMI cable from the Zotac into my TV, and I plugged a wired USB keyboard and mouse in the Zotac. I inserted the USB drive and booted it up. It automatically booted from the drive, and I selected Install.

Once the installation was finished, I had to go to Settings -> Sound and choose the HDMI output. Then, sound correctly played through my TV.

Setting up XMBC

There is a lot of software for home theater PCs. I chose XBMC since it seems very polished and has a lot of plugins for different video sources (such as Youtube). It’s also optimized for a remote instead of requiring a full keyboard.

I had to add an extra apt repository in order to install the latest version of XMBC (11.0 beta1). Here are the commands I ran:

sudo add-apt-repository ppa:nathan-renniewaldock/xbmc-stable
sudo apt-get update
sudo apt-get install xbmc

Then, I launched XBMC from Ubuntu. I was greeted with a screen that looked like:

I went to Settings and changed my region to where I live. I also told XBMC to update my library on start. Then, I went to Weather and entered my zip code to get local weather. Next I went to Videos -> Add ons and installed a bunch of video add ons (such as Youtube, HGTV, etc).

At this point, I had to manually run XBMC whenever I restarted the computer. There are many strategies to boot directly into XBMC. I decided to just add XBMC as a Startup Item in Ubuntu. I also allowed Ubuntu to boot without a password. So Ubuntu boots up and then runs XBMC when it’s finished. If I need to go back to Ubuntu, I can just exit from XBMC.

Uploading media

XMBC uses filenames to figure out which movie or TV show a file represents. For example, with default settings, episodes of a TV show need to be named in one of the following formats:

foo.s01e01.*
foo.s01.e01.*
foo.s01_e01.*
foo_[s01]_[e01]_*
foo.1x01.*
foo.101.*

These files should be in a folder with the name of the show. The full set of instructions can be found on the XBMC website at TV Shows (Video Library).

I created a Movies directory and a TV Shows directory under /home/paul/Videos and uploaded my media. Then, I added these directories to XMBC and set the content type to Movies and TV Shows respectively (see Media Sources).

Finally, I quit XBMC and restarted it. It scanned my directories and added the movies and TV shows to the library.

Hulu

Unfortunately, there are no officially supported Hulu plugins for XBMC. I managed to get an unoffical plugin working with some extra effort using Bluecop’s beta video plugin repository.

I downloaded the zip file from the site above and saved it to the Zotac. Then, in XBMC, I went to Settings -> Add ons and chose to install add ons from a zip file. At this point, the Hulu plugin appeared in the list of available plugins to install. I installed the plugin, but it still didn’t work.

By looking in the XBMC log (~/.xbmc/temp/xbmc.log), I discovered that Hulu requires a newer version of rtmpdump than Ubuntu ships with.

I built the latest rtmpdump from source with the following commands:

sudo apt-get install libssl-dev
git clone git://git.ffmpeg.org/rtmpdump
cd rtmpdump
make sys=posix
cp librtmp/librtmp.so.0 /usr/lib/i386-linux-gnu/librtmp.so.0

The last step copies the new librtmp on top of the existing one, which isn’t great. Unfortunately, XBMC runs with a specified library path, so I couldn’t find an easy way to add a different directory to the path.

Now, Hulu works. Unfortunately, I have to browse for videos to play. If I search for a video, and then play it, I get the following error in the xbmc.log:

ERROR: Error Type: <type 'exceptions.AttributeError'>
ERROR: Error Contents: _Info instance has no attribute 'videoid'
ERROR: Traceback (most recent call last):
                      File "/home/paul/.xbmc/addons/plugin.video.hulu/default.py", line 56, in <module>
                        modes ( )
                      File "/home/paul/.xbmc/addons/plugin.video.hulu/default.py", line 36, in modes
                        stream_media.Main()
                      File "/home/paul/.xbmc/addons/plugin.video.hulu/resources/lib/stream_hulu.py", line 148, in __init__
                        self.queueAD(video_id,2,1)
                      File "/home/paul/.xbmc/addons/plugin.video.hulu/resources/lib/stream_hulu.py", line 420, in queueAD
                        u += '&videoid="'+urllib.quote_plus(common.args.videoid)+'"'
                    AttributeError: _Info instance has no attribute 'videoid'

Browsing works just fine, though, so I only browse for videos.

Next Steps

I’m still using a wired USB keyboard and mouse, but I’m planning to get a home theater friendly wireless keyboard/mouse combo. Something along the lines of a Rii Mini Wireless Keyboard.

There are also hundreds of XBMC plugins I haven’t checked out yet, as well as other ways to customize XBMC.

We use foreman as our process launcher, but foreman does not handle running other projects in other rvm gemsets (and possibly ruby versions). One of my fellow developers, Tony Pitluga, released a gem called subcontractor that handles these cases.

For example, our Procfile looks like:

app: rails server
login: subcontract --rvm "--with-rubies rvmrc" --chdir ../login --signal INT -- rails server -p 4000

This runs the current app on port 3000 and the login app on port 4000 with a different home folder, ruby and gemset. The login line is roughly equivalent to running cd ../login && rvm --with-rubies rvmrc exec rails server -p 4000 The option --with-rubies rvmrc tells rvm to use the .rvmrc file to load the correct ruby and gemset. If a project doesn’t have an .rvmrc file, you can specify the ruby version:

another_app: subcontract --rvm ruby-1.8.7-p249@another_app --chdir ../another_app --signal INT -- rails server -p 4000

speclj comes with autotest built in. You can start an autotest process which will monitor the filesystem for changes. When tests or code change, the tests will be automatically run again. speclj-growl adds to this process by showing the test results in a growl popup, so you can stay within your editor, glance at the corner of the screen, and see whether your tests passed or failed.

For example, say you have a clojure project using both speclj and speclj-growl (in my example, the speclj-growl project itself). The command lein spec -a -f growl will start autotest (-a) and add the growl formatter (-f growl):

% lein spec -a -f growl

----- Tue Dec 06 23:03:32 CST 2011 -------------------------------------------------------------------
took 0.89041 seconds to determine file statuses.
reloading files:
  /Users/paul/speclj-growl/spec/speclj/report/growl_spec.clj

Growl Reporter
  report-runs
  - growls summary information for no test runs
  - growls a successful run
  - growls an unsuccessful run

Finished in 0.02542 seconds
3 examples, 0 failures

A growl message also appears on screen when the specs finish:

Now, if I change a spec to intentionally break it, the terminal automatically updates with the new spec output and a new growl message appears:

----- Tue Dec 06 23:04:58 CST 2011 -------------------------------------------------------------------
took 0.00414 seconds to determine file statuses.
reloading files:
  /Users/paul/speclj-growl/spec/speclj/report/growl_spec.clj

Growl Reporter
  report-runs  - growls summary information for no test runs (FAILED)
  - growls a successful run
  - growls an unsuccessful run

Failures:

  1) Growl Reporter report-runs growls summary information for no test runs     Expected: <"Failure">
          got: <"Success"> (using =)
     /Users/paul/speclj-growl/spec/speclj/report/growl_spec.clj:28

Finished in 0.00705 seconds
3 examples, 1 failures

This cycle continues every time the source or spec files change.

Check out the code and installation instructions on github: https://github.com/pgr0ss/speclj-growl

Thanks to Micah Martin for both speclj and help with speclj-growl.

For example, the Jukebox2 project depends on a bunch of java audio libraries like Jaad that do not live in a Maven repository. We tried checking these jars directly into the lib directory, but leiningen would delete these jars whenever we ran “lein deps.”

The solution we came up with was to make a local Maven repository within the project. Then, anyone can clone the project and run “lein deps” and get all of the dependencies, including the local jars. Here are the steps:

1. Create a directory in the project

   mkdir maven_repository

2. Add local jars to this repository

   For example, this command adds the jaad-0.8.3.jar file to the maven repository.

   mvn install:install-file -Dfile=jaad-0.8.3.jar -DartifactId=jaad -Dversion=0.8.3 -DgroupId=jaad -Dpackaging=jar -DlocalRepositoryPath=maven_repository

3. Add the following to project.clj

   :repositories {"local" ~(str (.toURI (java.io.File. "maven_repository")))}

4. Now a regular lein deps should work

   lein deps
    
   Downloading: jaad/jaad/0.8.3/jaad-0.8.3.pom from local
   Transferring 0K from local
   [WARNING] *** CHECKSUM FAILED - Error retrieving checksum file for jaad/jaad/0.8.3/jaad-0.8.3.pom - IGNORING

   The warning can be ignored, since the jar will be checked into the project and not downloaded from the internet.

The final results can be seen in the Jukebox2 project.

Thanks to Matjaz on the Leiningen mailing list for helping me figure out the right syntax for project.clj.

The problem is that most work done at companies do not have the same flow. For example, at Braintree Payments, we have about six developer pairs all working on different stories, all committing at the same time, and all with commit rights to our project. We trust that if a pair wrote the code, it’s good and is going into the project. In 99% of cases, if we are working on a feature, it is going to get released.

Given this kind of working environment, here is why I do not like feature branches:

Feature branches make for nasty commit history

Feature branching as a practice encourages lots of small commits, often before tests are passing or the code is in a useable state. Since these commits are on a branch, most people think it’s ok as long as the final merge commit back into master passes all of the tests.

The problem with this practice is that the commit history of the project is now full of broken work and non-passing code. I often use git history to get an idea of why a feature was done, or what other code was changed at the same time. I will “git blame” a method name to see when it was added, and what the code looked like at that point. With feature branches, I can no longer trust that commit. It may have been done on a branch, it may have broken half of the tests, and it may have been completely scrapped before the code went into master.

Furthermore, If feature branches are deleted once they are merged into master, where do bug fixes go? On a new feature branch or on master? There will no longer be an easy way to find a list of the commits that went into a feature. If feature branches are not deleted right away, the repository will likely end up with a huge number of branches.

Feature branches generally have no builds

We use Jenkins to continually test our application. We run numerous builds on every commit, including integration builds that call out to external dependencies.

When we do work on feature branches, we no longer have these builds. Developers will run the main suite of tests before they check in, but no one is going to run the full suite of integration tests on every commit. Furthermore, setting up new builds can be annoying, so we’re unlikely to do it unless we are going to maintain a branch for a long time.

Lots of branches become cumbersome

The lack of builds is one aspect of where lots of branches become cumbersome. Another is user testing. It is common to want the product owner or someone else to take a look at the feature before it’s done to get feedback. If you have a testing environment, they might want to deploy your code there to take a look (rather than looking over your shoulder on the workstation). But if you have feature branches, then they have to deploy your branch, which means they can no longer test other features at the same time. If they are in the middle of testing another feature that requires specific data, they may not be able to deploy your branch (e.g., if it has different database migrations that would require the database to be rebuilt).

Feature branches make refactoring harder

Modern version control tools like git make branching and merging much easier, but they are far from perfect. If we do large refactorings (especially refactorings that take numerous commits), it can become really hard to merge in feature branches. For example, if we rename methods, shift code between classes, or even reorder the methods in a class, the feature branch will likely get tons of conflicts on the merge. Then, someone has to go through each conflict and figure out where the code went. The longer the feature branch lives, the worse the merge becomes.

To take a specific example, say we want to switch our tests from one factory pattern to another. With feature branching, we would create a branch, and spend a few days porting over our tests. Then, when we merge back, we are going to run into conflicts with any tests that were changed by other feature branches. Even worse, while we are porting tests, other developers are still writing tests on their feature branches in the old style.

If we decide to make the test changes on master, but still have feature branches, all of the currently running feature branches have to continually merge from master in order to get the new tests. If they forget, and start modifying tests that have not been ported yet, they are still going to get conflicts when they merge back.

In contrast, if everyone is working in master, the people doing the refactoring can pull, make the changes, and commit frequently. As long as other developers also pull frequently (since everyone is making small, frequent commits), they automatically get the changes.

My preferences

Given that I don’t like feature branches, here is what I prefer:

All developers work in the master branch. When we are ready to release code to production, we merge master into a release branch, which gets some extra regression testing. We then tag from release and deploy the tag.

We use descriptive commit messages to signify what feature we are working on. For example, we use messages like “#card – message.” If we need to see all of the commits for a given feature, we can grep the git history for the card number. Likewise, if “git blame” a line of code, you can get back to the original story card and see why the feature was developed.

Commits only get pushed if they pass all of the tests. If a developer wants to make temp commits, that’s fine as long as they get squashed into a decent commit before pushing. Spike branches are fine for playing around, but the real work should be done on master once the approach is finalized.

On my last few projects, we have used a tool called rake_commit to do much of this work for us. It will prompt for a pair, story, and commit message, squash unpushed changes, run “git pull –rebase” (if working in git), run tests, and then push if the tests pass. This tool ensures a clean, linear git history.

For long running features, I prefer Branch by Abstraction and feature switches. This way, the features can be developed in master along side other development with the full suite of builds, but not necessarily impact the end user until the work is complete. If we have to release early for some reason (feature X just has to go out today), then we can release without the new features in use.

• Useful unix tricks
• Useful unix tricks – part 2
• Useful unix tricks – part 3

Use mount -o bind to remount a filesystem

When you mount a filesystem in linux, anything that was in the existing directory is now hidden and out of reach. For example:

% touch /mnt/foo
% ls /mnt
./  ../  foo

% sudo mount /dev/sdb1 /mnt
% ls /mnt
./  ../

If you want access to those files again, you can remount / using mount -o bind:

% sudo mkdir /mnt2
% sudo mount -o bind / /mnt2
% ls /mnt2/mnt/foo
./  ../  foo

This command remounts / under /mnt2, so you can access any file through /mnt2.

less -R shows colors for control characters

Programs like Rails write log output with terminal coloring information. If you open these files in less, you’ll see something like:

% less development.log

ESC[4;36;1mSQL (0.2ms)ESC[0m   ESC[0;1mSET client_min_messages TO 'panic'ESC[0m
ESC[4;35;1mSQL (0.1ms)ESC[0m   ESC[0mSET client_min_messages TO 'notice'ESC[0m

If you add the -R flag to less, it tells less to output raw control characters, which will color your terminal:

% less -R development.log
SQL (0.2ms)   SET client_min_messages TO 'panic'
SQL (0.1ms)   SET client_min_messages TO 'notice'

Pressing alt+. will put in the previous command's last argument

Pressing alt+. (the alt key and period) will drop the argument from the previous command into the terminal. For example:

% mv foo.txt bar.txt
% cat <press alt and .>

# this becomes
% cat bar.txt

This is especially useful for combo commands, such as unstaging and checking out a file in git:

% git st
# On branch master
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#       modified:   foo
#

% git reset HEAD foo
Unstaged changes after reset:
M       foo

% git co <alt+.>

This is part of the emacs key bindings, and you can read more about them on stack overflow: bash - first argument of the previous command

Show git branch info in the prompt with coloring

I do a lot of development with git, and I have a lot of different git repositories. It's handy to show git info in the prompt when I cd into a repo directory. At a minimum, I like seeing the current branch I have checked out.

For example, my prompt looks like:

host:~% cd repo
host:~/repo (master)%
host:~/repo (master)% git co -b some_branch
Switched to a new branch 'some_branch'
host:~/repo (some_branch)%

In zsh, getting a prompt like this is as simple as adding this to your .zshrc (or .zshenv) file:

git_prompt_info() {
  ref=$(/usr/local/bin/git symbolic-ref HEAD 2> /dev/null) || return
  echo " (${ref#refs/heads/})"
}

export PROMPT='%m:%~$(git_prompt_info)%# '

I like colors in my prompt as well, so my .zshenv looks like:

autoload -U colors
colors

git_prompt_info() {
  ref=$(/usr/local/bin/git symbolic-ref HEAD 2> /dev/null) || return
  echo " (${ref#refs/heads/})"
}

export PROMPT='%{$fg_bold[green]%}%m:%{$fg_bold[blue]%}%~%{$fg_bold[green]%}$(git_prompt_info)%{$reset_color%}%# '

After I load colors, I can use %{$fg_bold[color_name]} to change the following text to that color.

Use tmux instead of screen

For those that aren’t familiar with screen or tmux, here is the description from the tmux website:

tmux is a terminal multiplexer: it enables a number of terminals (or windows), each running a separate program, to be created, accessed, and controlled from a single screen. tmux may be detached from a screen and continue running in the background, then later reattached.

Being able to detach and reattach is really nice when working on remote servers. You can ssh to a server, start tmux, run a long running process and then detach. Hours later, you can ssh to the server again and reattach to the tmux window. The program continued running after you detached, and you can see the output as if you never left.

tmux is also great for working with others remotely. Everyone can ssh into one server and connect to the same tmux. Then, everyone can see the same file, commands, etc. This is often how we work together on deploys at Braintree Payments.

If you currently use screen, considering switching to tmux. They are very similar (both support everything I mentioned above), but tmux adds a few nice features:

1. tmux allows you to split the screen vertically so you can have two windows side by side. This is great for working on servers. You can tail the log in one split, and work in the other so you can keep an eye on the logs. Screen lets you split horizontally, but I find vertical splits more useful on widescreen monitors since there is not as much vertical real estate.
2. tmux adds a customizable status bar to the bottom of the terminal with information like which server you are on, the current time, tabs, etc.
3. If different people all connect to the same tmux, it will adjust the viewable area to be the smallest window size. Larger monitors will see a grayed out background filling in the extra space. In contrast, screen might put output where the smaller windows cannot see it.

column -t will reformat input into columns

Use column -t when printing tabular output where the columns do not align. For example:

% print "one two\nthree four"
one two
three four

% print "one two\nthree four" | column -t
one    two
three  four

I use this all the time with the program pg_lsclusters, which prints out postgresql cluster information:

% pg_lsclusters
Version Cluster   Port Status Owner    Data directory                     Log file
8.4     backup_main 5433 down   postgres /var/lib/postgresql/8.4/backup_main /var/log/postgresql/postgresql-8.4-backup_main.log
8.4     main      5432 online postgres /var/lib/postgresql/8.4/main       /var/log/postgresql/postgresql-8.4-main.log

% pg_lsclusters | column -t
Version  Cluster      Port  Status  Owner     Data                                 directory                                           Log  file
8.4      backup_main  5433  down    postgres  /var/lib/postgresql/8.4/backup_main  /var/log/postgresql/postgresql-8.4-backup_main.log
8.4      main         5432  online  postgres  /var/lib/postgresql/8.4/main         /var/log/postgresql/postgresql-8.4-main.log

ctrl+p and ctrl+n cycle through terminal history

If you want to run a previous command in terminal, it’s common to press the up and down arrows to cycle through old commands. You can also use ctrl+p for previous and ctrl+n for next. I prefer these keys since it leaves my hands near the home row rather than having to reach for the arrow keys.

While it was still running fine for me, I wanted to make changes to my blog, and I figured it was time to move to software that is still active.

After searching around, I decided to move to WordPress. My main reasons were:

1. Existing script to migrate all of my posts and comments from Mephisto.
2. Lots and lots of useful plugins.
3. Widely used and actively developed.

I used a project on github called mephisto-to-wxr to export my Mephisto content as a WordPress export file. The script was a little outdated, so I forked and got it working with with my setup. You can check out my fork at: https://github.com/pgr0ss/mephisto-to-wxr

The syntax highlighting plugin I used with Mephisto required tags around code that looked like:

<pre><code class="ruby">
</code></pre>

In WordPress, I’m using a plugin called WP-Syntax which requires tags like:

<pre lang="ruby">
</pre>

To get the required output, I ran the WordPress export (which is xml) through the following sed command:

sed 's/<pre><code class=/<pre lang=/g' pgrs.net.wxr | sed 's/<\/code><\/pre>/<\/pre>/g' > pgrs.net.syntax_fix.wxr

For those interested, here is my current setup in WordPress:

• Suffusion theme
• Plugins:
  • WordPress Importer to import the exported wxr file
  • Akismet to prevent comment spam
  • About Me 3000 for the About Me in the upper right
  • WP Super Cache for page caching
  • WP Syntax for syntax highlighting