John Barnette

Isolate on Heroku

2010-06-26T00:00:00-07:00

Update: Jamie Macey wrote a gem for this, and we improved the logic a bit. Use it.

Quite a few people have asked about using Isolate on Heroku. There’s no built-in support for Isolate in Heroku’s current deployment tools, but it’s pretty easy to hack together using Heroku’s support for a .gems manifest file.

Assumptions

You’re using some sort of rake release task for deployment. See “Alternatives” at the end of this post for, uh, an alternative.
You’re using Isolate with its :system option set to true. If you’re using Isolate 2, this is the default. :system => true allows isolated projects to use gems that are already installed on the system, which is necessary for Heroku support.
I’m testing this with a simple isolated Rails 2.3.8 app, but the process is basically the same for any Rails version (or other framework). See the examples in Isolate’s README for details.

Dot Gems

Heroku’s .gems manifest is a simple text file containing one gem dependency per line, in a format that’s very similar to a gem command line. Heroku will read this file every time you release/push reinstall all your gem dependencies if it’s changed.

We’re going to generate .gems from our Isolate file. Here’s my lib/tasks/isolate.rake file:

desc "Generate a .gems manifest."
task :dotgems => %w(Isolate lib/tasks/isolate.rake) do
  File.open ".gems", "wb" do |f|
    entries = Isolate.sandbox.entries.sort_by { |e| e.name }

    f.puts "isolate --version '= #{Isolate::VERSION}'"

    entries.each do |entry|
      next unless entry.matches? "production"

      gems  = [entry.name]
      gems << "--version '#{entry.requirement}'"

      if entry.options[:source]
        gems << "--source #{entry.options[:source]}"
      end

      f.puts gems.join(" ")
    end
  end

  abort "Commit .gems, it changed." if /\.gems/ =~ `git status`
end

This task looks at all your isolated gem entries, picks the ones that are valid for production, and writes each of them to the .gems file. It also adds Isolate itself as a dependency. If .gems has changed, the task aborts and reminds you to commit the changes.

Releasing

My rake release task does a fair amount of stuff before it pushes to Heroku, like creating a release email and running some sanity checks. To make sure my gem dependencies are always up-to-date, I just make sure the release task depends on dotgems:

task :release => :dotgems

Done. Any time I run rake release, .gems will either be up-to-date or prompt me to commit the changes before continuing.

Alternatives

This works Well Enough for me at the moment, but it could certainly be smoother. If you deploy using git push instead of wrapping it in a Rake tasks, you could use a Git pre-commit hook to regenerate .gems if necessary.

Isolate 2 has added some lifecycle event hooks, so it’d also be pretty easy to autogenerate .gems every time Isolate runs.

Of course, the best possible solution would be for Heroku to support Isolate natively, but I wouldn’t hold your breath for that. :-)

Deploying to Heroku

2009-11-10T00:00:00-08:00

Deploying to Heroku with git push is awesome. I’m running a couple of different environments, though, and there’s extra stuff that I want to do when I deploy. Rake to the rescue!

Some Assumptions

Okay, here’s how I roll: origin is GitHub. origin/master is the production branch, not the head of development. origin/next is, well, what’s next (you probably call it “staging”). There are local tracking branches for each of these, natch.

New features happen in topic branches, and they’re merged into next when they’re ready. Emergencies and bug fixes are similar, but they also get merged into master and deployed before next is fully baked.

master is only ever updated by bug/emergency branch merges and periodic merges of next. If git merge next isn’t a fast-forward, something’s probably wrong.

There are (at the moment) two Heroku environments, myapp-next and myapp-master. There’s a Git remote for each environment:

$ git remote
heroku-next
origin
heroku-master

Got it? Good.

Deploying

$ rake deploy

Super complicated. In my setup, this task…

Complains if I’m trying to deploy from a branch that doesn’t match an deployment env (deploys can only happen from master and next),
sets the TO environment variable for Hoptoad,
shows a list of all the changes since the last deploy, and how long it’s been,
pushes the current branch to the matching remote, e.g., git push heroku-next next:master,
tells Hoptoad about the deploy, and
sends a fancy email w/changes to the deployment mailing list.

The First Time

$ rake deploy:newb

This task installs the heroku and taps gems if they’re not available, creates and fetches the Heroku remotes for each environment, and makes sure you have local tracking branches for the origin/bleh equivalents of each remote.

What changed?

$ rake deploy:pending

This task shows a list of commits that have happened since the last deploy. It’s a pretty thin wrapper around git log.

I Hate Conclusions

There are a few other lifecycle hooks for attaching stuff that needs to happen before or after the deploy, but that’s pretty much it. A lightly edited version of my lib/tasks/deploy.rake is available in this Gist. Add to it, correct it, abuse it.

Boring Things First

2009-09-07T00:00:00-07:00

This is a draft.

Yeah, yeah. You’re agile. You always test first, you work in tight iterations, you have YAGNI tattooed on your forehead. Good for you. Guess what? You’re building on sand. Do this stuff first:

Tests (and Autotest) With Coverage

Make sure your testing infrastructure works. Write a single sanity test, make it pass and fail, and make sure that your test scripts/tasks, autotest setup, and coverage tools are working correctly.

Deploy/Install/Update an Empty App

Before you write a damn thing, make sure you can distribute. If you’re working on a Rails app, get your Vlad or Capistrano setup tuned and perfect. Do a bunch of blank deploys.

If you’re writing a desktop app, write your installer and make sure it’s working. Don’t wait ‘til the last minute! The moment you write a line of code, it’ll be easier to have someone test it if the installer’s functional.

Even better, start by integrating an update system like Sparkle. You may be distributing the empty shell of an app, but now you can automatically update your testers every time you push new changes. And you’re testing your post-release update infrastructure at the same time! Win!

Error Notification

Some of your code is shit. Even in the unlikely event that all your Codes Are Perfect, someone’s code that you depend on is shit. Integrate error reporting at the very beginning, before any users have a chance to make errors. For Rails apps, I use Hoptoad. There are a billion other alternatives, and rolling your own is easy too.

Fundamental Docs

You’re probably not going to be the only person who works on this project. How does the next person in the door get up and running? If the answer is much more complicated than “check out the code and read the README,” you suck.

Start documenting at the very beginning. What should someone run to get started in a fresh checkout? (I like rake newb) How does deployment work? Where’s the deployment server? How was it configured? How do I stand up another one from scratch? What external services (issue tracking, error reporting) does this app use?

Make sure your initial set of docs answers these questions, and make it clear that keeping them up-to-date is as important as keeping the tests passing.

On Rake

2009-08-27T00:00:00-07:00

I’ve been reading a lot of Rakefiles lately, and it’s obvious that the folks writing them think of Rake as An Engine For Encapsulating Tasks. That’s fine, but it’s only part of what Rake can do.

I Do Declare

Rake is all about resolving dependencies. It’s about being declarative, not imperative. Especially when dealing with files and directories, this can make a huge difference. Here’s an example of a imperative task:

desc "Make sure your database.yml is in place"
task :wheres_your_database_yml do
  unless File.exists? File.join("config", "database.yml")
    puts "Where's your database.yml, dude?"
    if File.exist? File.join("config", "database.yml.example")
      puts "(elided)... so I'll make a copy of it for you."

      cp File.join("config", "database.yml.example"),
        File.join("config", "database.yml")

        abort "(elided)...rerun the last command."
    else
      abort "(elided)...There's no config/database.yml.example..."
    end
  end
end

(This is from technicalpickles’ dude-wheres-your-database-yml, a nice little Rails plugin for automatically installing a database.yml file the first time a new contributor runs Rake. I’ve shortened a few of the messages, but that’s it.)

This says, “Hey, if there’s not a config/database.yml file, create it by copying config/database.yml.example. If THAT doesn’t exist, complain.”

Here’s how to write this in a more declarative way:

file "config/database.yml" => "config/database.yml.example" do |t|
  sh "cp #{t.prerequisites.first} #{t.name}"
end

(This doesn’t abort or talk to the user, but you get the idea.)

This says essentially the same thing. file declarations in Rake behave just like tasks – they can have prerequisites, be dependencies, and be invoked via rake – but they only run if the target doesn’t exist or the prerequisites are newer than the target.

There’s also a shortcut for creating directories:

directory "foo"

# ...is the same as:
file("foo") { |t| mkdir t.name  }

It’s also possible to declare a transformation for a set of files, not just a single file. Let’s say that you have a directory of Markdown files, and you want to be able to transform any of ‘em into HTML. Easy:

rule ".html" => ".markdown" do |t|
  sh "markdown #{t.source} > #{t.name}"
end

# in your shell
$ rake foo.html

Yay! Again, this will only run when the dependencies are unresolved, that is, if the target is missing or the source is newer.

Picking Nits

A grab-bag of other bad usage I’ve seen recently:

Prerequisites

task :mystuff do
  # ...
end

task :something => :mystuff

This isn’t necessary. Unless you want to be able to use your mystuff task independently, just define something again:

task :something do
  # my stuff...
end

“Redefining” an existing task in Rake appends the new actions (and prerequisites) to the original. Don’t add new tasks unless you actually need them.

Explicit Invocation

Sometimes it’s necessary to run Rake tasks imperatively. That’s fine. Run them with invoke, not with execute:

task :mine do
  # some setup stuff
  Rake::Task["theirstuff"].invoke
end

invoke uses the same rules as normal Rake execution: If the task has run already, it won’t run again. execute runs the task no matter what. Make sure you know which one you want.

Fin

For a more detailed explanation of directory, file, and rule, take a look at the Rake documentation.

Environment Setup

2009-07-24T00:00:00-07:00

Note: I’m rereading this in the first quarter of 2011 and it’s pretty obsolete. I’ll update it at some point.

I’ve had scattered notes on setting up a new development computer from scratch in Backpack for years now. Hopefully collecting them here will make it more likely that they’ll stay up-to-date. This is my environment, probably not what I’d suggest for you.

I don’t necessarily install all of these all the time (MySQL), but I’d still rather have all the instructions in one place.

These notes assume Mac OS X Leopard, with the developer tools installed.

Fundamental Stuff

1Password. It knows all.
Dropbox. Sync folders transparently.
Emacs. I’m building Emacs 23 via MacPorts.
Inconsolata. My favorite programming font.
LaunchBar. Useless without it.
MacPorts. I use this to manage as much dev stuff as I can.
Things. What am I supposed to be doing right now?

Important Stuff

Adium. I like talking on the digital internets.
FireFox. I use Safari for daily browsing, but FF for lots of dev.
Growl. I turn off most notifications, but it’s still good to have.
Mail Act-On. Effortless filing in Mail.app.
Mail Unread Menu. Since I hide the dock and mute everything.
Parallels. Only for testing stuff on Windows.
Shimo. VPNs are dumb, but Shimo is pretty.
Transmit. FTP, SFTP, S3.
Tweetie. WHAT ARE YOU DOING?!
Xmarks. Cross-browser, cross-machine bookmark syncing.

Nice Stuff

FuzzyClock. Normal clocks are boring.
GitNub. Good for quick repo history scans.
Skitch. Screenshots + sharing.

Profile, Emacs Configuration, etc.

I keep most of my dotfiles in a Git repository and symlink them. I do the same thing with Dropbox for 1Password and Things.

RubyGems

$ sudo gem update --system

Git

$ sudo port install git-core +svn +doc
$ git config --global user.name John Barnette
$ git config --global user.email jbarnette@gmail.com

MySQL

$ sudo port install mysql5-server
$ sudo mysql_install_db5 --user=mysql
$ sudo port load mysql5-server

Add /opt/local/lib/mysql5/bin to your path, and this should work:

$ mysql -uroot

MySQL Ruby Bindings

sudo gem install mysql --                                    \
  --with-mysql-include=/opt/local/include/mysql5             \
  --with-mysql-lib=/opt/local/lib/mysql5                     \
  --with-mysql-config=/opt/local/lib/mysql5/bin/mysql_config

Versioning HTTP APIs

2009-04-07T00:00:00-07:00

There’s a lot of hot air circulating online re versioning HTTP APIs. I ignore most of it: It’s RESTy folks without real load or customers, wanking about purity.

I actually agree with some of the purist noise from a theoretical perspective, but in practice the implementations are annoying enough that they’re simply not worth the effort.

These suggestions apply to external APIs: If you control all the endpoints, or if you’re running a service where folks are more accepting of rapid change, you don’t need to worry nearly as much.

These also apply only to systems where there are lots of clients and a single service endpoint. The landscape changes a bit if clients are hitting multiple, independent installations of your service.

My rules of thumb for versioning an HTTP API:

Do Your Best To Never, Ever, Break Compatibility

You don’t need new versions if you don’t break client compatibility! Don’t make casual changes to the API. Remember your consumers.

Don’t change resource URLs without a damn good reason.
Don’t let attribute/parameter positions matter.
Accept and ignore unknown attributes/parameters.

If you follow these guidelines, there’s no reason to break major API compatibility when you add new resources, parameters, or attributes.

Admit That You’re Going to Break Compatibility

Nobody’s perfect. It’s gonna happen. Acknowledge it from the beginning, and design your URL scheme accordingly:

http://example.com/api/v1/widgets.json
http://example.com/api/v2/widgets.json

Many RESTy folks object to this sort of namespace segmentation, because a resource should only have one identifier. That makes a lot of sense, but the implementation difficulties outweigh the benefits. It’s a lot easier to route completely separate namespaces than it is to use content negotiation or any of the other “more proper” solutions.

Pick Sunsets and Stick To Them

How many API versions are you going to support? For how long? Be firm. I haven’t generally seen a need to support more than two versions of an API simultaneously.

Be helpful when you kill an API version. Have any requests to the old namespace return good error codes and messages (or permanent redirects to the new API).

Be aggressive about communicating with your API consumers. Don’t let them get caught by surprise when an API version goes away.

Write Documentation as You Go

Most APIs start as an internal tool, and then get exposed to the world later. Most internal tools are very lightly documented. Don’t do that! If it’s an interface, even if it’s currently only used internally, document it as if it’s intended for a wider audience. If you document incrementally, the burden is very low.

Write Really Damn Good Integration Tests

Keep your coverage high. Don’t ever change tests, except to fix a test bug. You can add to them, but if you’re changing them you’ve probably broken compatibility. When you have to break compatibility, copy the current tests, document the difference in the test and the documentation, and alter them for the new version.

Ship a Service Client

Along with good service documentation, try to ship at least one client implementation of your API. If it’s good, clean, well-tested, and well-documented, folks will use it as a guide when they port to their preferred environment.

Design your API via working tests and it’ll be more usable in the long run. If it’s possible, have different people implement the server and the client.

Separate API from UI

If you’re developing in a framework like Rails, avoid the temptation to mix your API and your UI controllers. Work hard to keep them separated. If there’s enough logic in your UI controllers that it’s hard to break out the API bits, your controllers are too fat. Refactor.

A Quick (Rails) Example

Here’s how I might implement a (very) small API. We’ll start with a single resource – a widget – and build from there.

# Api::V1::WidgetsController
controllers/api/v1/widgets_controller.rb

# Api::V1::WidgetsControllerTest
test/functional/api/v1/widgets_controller_test.rb

# Api::V1::WidgetsTest
test/integration/api/v1/widgets_test.rb

# in routes.rb
map.namespace :api do |api|
  api.namespace :v1 do |v1|
    v1.resources :widgets
  end
end

Adding an Attribute to the Widget Model

No new version is necessary: just add it. Don’t forget to change the docs! Remember to clearly document that clients, like the server, should ignore unknown attributes.

Adding a New Resource

No new version is necessary: just add it. If the clients don’t know about a resource, they don’t need to care. See the next section if you’re adding a relationship to an existing resource: The same restrictions apply.

Renaming an Attribute on the Widget Model

Okay, so this is a breaking change. Time for v2!

$ cp -r app/controllers/api/v1 app/controllers/api/v2
$ cp -r test/functional/api/v1 test/functional/api/v2
$ cp -r test/integration/api/v1 test/integration/api/v2
$ cp -r doc/api/v1 doc/api/v2

Use the method of your choice to change all references under the new paths from “V1” to “V2”.

Don’t forget to copy your routes to a new v2 namespace. Once you’ve done all this for the first time, write a Rake task to do it for you. For extra points, write a task that removes an older API version and make the generation task complain if you’re trying to support too many version simultaneously.

Add your breaking features, change the new v2 tests, update the docs and your client reference implementation, and get on with your life.

When the new version goes live, it’s time to start communicating with your consumers about when the old version is going away.

Focusing Autotest

2009-01-23T00:00:00-08:00

Looking at micronaut made me realize that I often want to iterate very quickly on a single test, rather than having Autotest re-run every test in the file each time I save. After a quick chat with Ryan – whose suggestion was way simpler than what I was considering – here’s what popped out:

class Test::Unit::TestCase
  def self.focus *focused
    focused.collect! { |m| m.to_s }

    instance_methods.
      collect { |m| m.to_s }.
      select  { |m| m =~ /^test/ }.
      reject  { |m| focused.include? m }.
      each    { |m| undef_method m }
  end
end

With this snippet in my test helper file, I can easily focus on just one test:

class MyTest < Test::Unit::TestCase
  # lots of tests

  def test_thingy
    # ...
  end

  # lots more tests

  focus :test_thingy
end

…and remove the focus when I’m ready to zoom out. Awesome. Sprinkle on some editor support and it’ll be epic.

(2009-03-27: This is available in the latest ZenTest.)

Parsing Ruby is Hard

2009-01-22T00:00:00-08:00

"Parsing #{"Ruby #{"is #{ %(hard)}"}"}."
# => "Parsing Ruby is hard."

Note to Self

2009-01-21T00:00:00-08:00

Be Confident

Tests build confidence. Write ‘em. They’ll save your ass, and they’ll let you take a chainsaw to your code without being afraid of unintended consequences.

Be Lazy

Write tools. If it’s happened more than twice, don’t ever do it by hand again.

Be Asynchronous

If it can be done outside the request/response cycle, consider queuing it. Mailers, uploads, audit trails, anything with an external system dependency or a lot of IO.

Be Stateful

If there’s a lifecycle, model it as a real state machine. Beware ad hoc flags.

Be Clear

You’ll write it once, but you’ll read it a lot. Code accordingly. Sometimes simplicity takes a bit longer, but it’ll pay off.

Be Consistent

Inconsistent file names, task names, and coding styles hurt productivity.

Be Timely (but not too timely)

Keep frameworks, plugins, libraries, and tools up-to-date, but think twice before using a production app to play with the bleeding edge.

Be Certain

Don’t speculate, get data. Act on what you know, not what you suspect. Is that code really faster? Do users really want that feature?

Be Persistent

Find the root cause. Keep asking why, even when you’re tired and under the gun. The guesswork patch you write today will be a nightmare tomorrow.

Be Wrong

If it’s not working, change it, no matter how long it took to write. Don’t throw good money after bad. Admit mistakes early and often.