Digressions of a Drive-by Hacker

CSSpring Cleaning

2009-09-08T00:00:00-07:00

CSSpring Cleaning

I was talking to some front-end devs a couple weeks ago about maintaining CSS on an evolving website. In particular, how to figure out which rules are no longer being applied (remnants from experiments / old designs / whatever) and remove them.

I remembered seeing something a few months ago that claimed to do this. After digging around on GitHub for a bit, I re-found deadweight. Unfortunately, after looking at it briefly, I realized that it made some pretty strong assumptions, mainly about being run in the context of a Rails app.

Time for some hacking.

My fork can be installed with RubyGems (0.0.5 is required for deflate and gzip content-encodings):

$ sudo gem install mojodna-deadweight -s http://gems.github.com/

The first step was to make it accessible as a command-line utility (ideally in the spirit of the UNIX Philosophy).

$ deadweight -s styles.css -s ie.css index.html about.html

(This will check styles.css and ie.css against index.html and about.html, outputting unused rules.)

It also supports input from pipes, meaning that you can chain it together and filter orphaned rules without writing them to a file (you’ll have to configure logging in order to limit output).

$ cat styles.css | deadweight index.html

Deadweight now contains an experimental -L argument that causes it to use Lyndon if the lyndon executable is in your $PATH (only possible on OS X with MacRuby installed). It was a bit flaky in my testing, but I may be using an older version of MacRuby. Great idea though. Check this post for more info on what it does / how it works.

The next step was to expose it as an HTTP proxy. It uses 8002 by default for no particular reason. (oauth-proxy uses 8001.)

$ deadweight -l deadweight.log -s styles.css -w http://github.com/ -P

(This dumps log output to deadweight.log in $PWD and matches http://github.com/* against styles.css.)

It’s probably most useful as an HTTP proxy; you can start it with a list of target stylesheets, configure your browser to load pages through it (sorry, it gets really slow when requesting gzip- or deflate-encoded pages, which is a lot of the time), and watch the output to see how many rules it’s hit. When you’re done, hit ^C to stop it and (optionally) output the remaining CSS rules into a file for you to examine (-o orphans.css will do this for you when it shuts down).

If you’re lucky, the file containing orphaned CSS will be empty. If not, re-start the proxy with the orphaned CSS as the target and continue browsing until you’re ready to start looking for matches by hand (this is necessary, as it won’t catch classes applied by JavaScript).

Things to Improve

WEBrick’s HTTP proxy implementation leaves a lot to be desired (it’s nowhere on the same level as Twisted’s), but it’s the only game in town. Ilya Grigorik has done some excellent work with EventMachine and proxies, but it doesn’t appear that anyone has built a general-purpose HTTP proxy that easily supports header modification. It’s not a particularly hard problem and being able to move deadweight’s proxy implementation to such a base would provide the ability to turn off compression and thus speed things up considerably.

One could also imagine a world in which the proxy server also contains a web interface that displays unmatched CSS rules and updates on the fly. EventMachine would do the trick here, too.

Exploring OAuth-Protected APIs

2009-08-21T00:00:00-07:00

Exploring OAuth-Protected APIs

From time to time I need to debug OAuth-protected APIs, checking response headers and examining XML and JSON payloads. curl generally rocks for this sort of thing, but when the APIs in question are protected with OAuth, things break down. Likewise for benchmarking (ab, httperf, etc.) and exploration–isn’t it nice to browse APIs that return XML in Firefox?

This needn’t to be the case.

Enter `oauth-proxy`

This is why I wrote oauth-proxy. It does what it says on the tin: it acts a proxy server that transparently adds OAuth headers to requests.

There are 3 steps to using it. First, clone the repository:

$ git clone git://github.com/mojodna/oauth-proxy.git

Then, start it:

$ cd oauth-proxy
$ twistd -n oauth_proxy \
    --consumer-key <consumer key> \
    --consumer-secret <consumer secret> \
    [--token <token>] \
    [--token-secret <token secret>] \
    [-p <proxy port>] \
    [--ssl]

If you’re accessing a resource that only requires 2-legged OAuth, you can omit --token and --token-secret. The proxy port defaults to 8001, and --ssl is used if you’re proxying connections to a provider that requires SSL (Fire Eagle, for example).

In order to run the proxy, you’ll need Python and a modern version of Twisted (I use 8.2.0). OS X Leopard comes with Python 2.5.1 (which is sufficient) and an old version of Twisted (2.5.0), so you’ll need to update that:

$ easy_install twisted

Once it’s been started, use curl to make requests through it:

$ curl -x localhost:8001 http://host.name/path

You can also benchmark your APIs through it using ApacheBench (ab, as it includes support for HTTP proxies). Note that you are introducing additional overhead by proxying the request, so your numbers may be a bit off.

$ ab -X localhost:8001 http://host.name/path

Firefox (and browsers in general) supports HTTP proxies, so you can add a “Manual Proxy Configuration” and pass requests through oauth-proxy to explore APIs in the comfort of your favorite browser. By default, all requests that Firefox makes will go through the proxy and be signed, which may confuse other websites you visit (and will inadvertently reveal your consumer key and access token but not the corresponding secrets).

Access Tokens Sold Separately

If you’re accessing a resource that requires 3-legged OAuth, you’ll need a token. You may already have one, but if you don’t, you can use the OAuth library for Ruby to obtain one.

First, install the gem (0.3.5 is current as of this writing):

$ sudo gem install oauth

Then, trigger the authorization process from the command-line:

$ oauth \
    --consumer-key <consumer key> \
    --consumer-secret <consumer secret> \
    --access-token-url http://host.name/path/to/access_token \
    --authorize-url http://host.name/path/to/authorize \
    --request-token-url http://host.name/path/to/request_token \
    authorize

Follow the prompts, and voilà, an access token and secret that you can use with oauth-proxy.

A Concrete Example

Twitter’s popular, right? Let’s use that.

First, register an application to get a consumer key and secret. I registered as a “client” application, since the command-line still doesn’t have a callback url.

Once that’s done, you’ll get a set of credentials that can be used to initiate the authorization process.

Let’s authorize.

$ oauth \
  --consumer-key <consumer key> \
  --consumer-secret <consumer secret> \
  --access-token-url http://twitter.com/oauth/access_token \
  --authorize-url http://twitter.com/oauth/authorize \
  --request-token-url http://twitter.com/oauth/request_token \
  authorize

After following the prompts, you’ll get something back that looks like this:

oauth_token_secret: [redacted]
oauth_token: [redacted]
user_id: [redacted]
screen_name: [redacted]

You can then use those values to start the OAuth proxy:

$ twistd -n oauth_proxy \
    --consumer-key <consumer key> \
    --consumer-secret <consumer secret> \
    --token <access token> \
    --token-secret <token secret>

Now we’re set. Let’s go exploring:

$ curl -sx http://localhost:8001 \
    http://twitter.com/statuses/friends_timeline.json | \
    jsonpretty | pygmentize -l js

You’ll get exactly what you’re expecting and you’ll be using OAuth (this is a partially contrived example since Twitter still supports HTTP Base Auth).

(jsonpretty rocks. pygmentize (easy install pygments) makes it easier to make sense of the chaos.)

That’s all. I use this stuff all the time and can’t imagine debugging APIs without it.

Subscribing to Wordpress.com with Switchboard

2009-07-21T00:00:00-07:00

Subscribing to Wordpress.com with Switchboard

Last week, Andy Skelton announced Real-time Wordpress.com subscriptions via XMPP. It does IM, but more interestingly for me, it supports PubSub. Had I realized this when he posted that, I would have included it as an example in switchboard : XMPP :: curl : HTTP alongside Fire Eagle and Superfeedr. What’s done is done, however, so here’s a quick rundown of how to interact with Wordpress.com using Switchboard.

First, let’s get a basic idea of the identities and features that Wordpress.com advertises:

$ switchboard disco --target pubsub.im.wordpress.com info
=> Switchboard started.
Discovery Info for pubsub.im.wordpress.com
Identities:
  pubsub/service: Publish-Subscribe

Features:
  http://jabber.org/protocol/disco#info
  http://jabber.org/protocol/disco#items
  http://jabber.org/protocol/pubsub
  vcard-temp
  http://jabber.org/protocol/pubsub#access-authorize
  http://jabber.org/protocol/pubsub#access-open
  http://jabber.org/protocol/pubsub#access-presence
  http://jabber.org/protocol/pubsub#access-whitelist
  http://jabber.org/protocol/pubsub#auto-create
  http://jabber.org/protocol/pubsub#collections
  http://jabber.org/protocol/pubsub#config-node
  http://jabber.org/protocol/pubsub#create-and-configure
  http://jabber.org/protocol/pubsub#create-nodes
  http://jabber.org/protocol/pubsub#delete-items
  http://jabber.org/protocol/pubsub#delete-nodes
  http://jabber.org/protocol/pubsub#instant-nodes
  http://jabber.org/protocol/pubsub#item-ids
  http://jabber.org/protocol/pubsub#last-published
  http://jabber.org/protocol/pubsub#manage-subscriptions
  http://jabber.org/protocol/pubsub#member-affiliation
  http://jabber.org/protocol/pubsub#modify-affiliations
  http://jabber.org/protocol/pubsub#multi-subscribe
  http://jabber.org/protocol/pubsub#outcast-affiliation
  http://jabber.org/protocol/pubsub#persistent-items
  http://jabber.org/protocol/pubsub#presence-notifications
  http://jabber.org/protocol/pubsub#presence-subscribe
  http://jabber.org/protocol/pubsub#publish
  http://jabber.org/protocol/pubsub#publisher-affiliation
  http://jabber.org/protocol/pubsub#purge-nodes
  http://jabber.org/protocol/pubsub#retract-items
  http://jabber.org/protocol/pubsub#retrieve-affiliations
  http://jabber.org/protocol/pubsub#retrieve-default
  http://jabber.org/protocol/pubsub#retrieve-items
  http://jabber.org/protocol/pubsub#retrieve-subscriptions
  http://jabber.org/protocol/pubsub#subscribe
  http://jabber.org/protocol/pubsub#subscription-notifications
  http://jabber.org/protocol/pubsub#subscription-options
Shutdown initiated.

Lo, it supports PubSub! (But it doesn’t support node discovery, so we can’t get a list of available nodes.) Fortunately, Andy’s post includes a couple. Let’s subscribe:

$ switchboard pubsub --server pubsub.im.wordpress.com \
    --node /blogs/andy.wordpress.com subscribe
=> Switchboard started.
Subscribe successful.
Shutdown initiated.

$ switchboard pubsub --server pubsub.im.wordpress.com \
    --node /blogs/andy.wordpress.com/comments subscribe
=> Switchboard started.
Subscribe successful.
Shutdown initiated.

$ switchboard pubsub --server pubsub.im.wordpress.com --node \
     /blogs/andy.wordpress.com/2009/07/16/real-time-wordpress-com-subscription/ \
    subscribe
=> Switchboard started.
Subscribe successful.
Shutdown initiated.

Switchboard claimed that that worked, but let’s double-check and list our subscriptions:

$ switchboard pubsub --server pubsub.im.wordpress.com subscriptions
=> Switchboard started.
Subscriptions:
4E039BC19E028: me@jabber.org => /blogs/andy.wordpress.com (subscribed)
4E039D2A2B691: me@jabber.org => /blogs/andy.wordpress.com/2009/07/16/real-time-wordpress-com-subscription (subscribed)
4E039BC7E8A69: me@jabber.org => /blogs/andy.wordpress.com/comments (subscribed)
Shutdown initiated.

Actually, there was no need to subscribe to comments for “Real-time Wordpress.com subscription” because we’re also subscribed to the comments feed. Let’s unsubscribe and check our subscriptions again:

$ switchboard pubsub --server pubsub.im.wordpress.com --node \
     /blogs/andy.wordpress.com/2009/07/16/real-time-wordpress-com-subscription \
    unsubscribe
=> Switchboard started.
Unsubscribe was successful.
Shutdown initiated.

$ switchboard pubsub --server pubsub.im.wordpress.com subscriptions
=> Switchboard started.
Subscriptions:
4E039BC19E028: me@jabber.org => /blogs/andy.wordpress.com (subscribed)
4E039BC7E8A69: me@jabber.org => /blogs/andy.wordpress.com/comments (subscribed)
Shutdown initiated.

Ok, all good. Now let’s hope Andy posts something or gets a comment on something so we can listen for it:

$ switchboard pubsub --server pubsub.im.wordpress.com listen
(sample post and comment <event /> stanzas to come)

For bonus points, let’s implement a WordpressJack:

class WordpressJack
  def self.connect(switchboard, settings)
    switchboard.plug!(AutoAcceptJack, NotifyJack, PubSubJack)
    switchboard.hook(:new_post, :new_comment)

    switchboard.on_pubsub_event do |event|
      event.payload.each do |payload|
        payload.elements.each do |item|
          # grab the feed out of the payload
          feed = item.first_element("feed")
          
          # TODO determine whether it's a comment or a post
          # for now, we'll assume it's a post
          # invoke the :new_post hook with the feed subtree
          # (later: parse it with FeedTools or something)
          on(:new_post, feed)
        end
      end
    end
  end
end

And a quick-and-dirty consumer:

#!/usr/bin/env ruby -rubygems

require 'switchboard'
require 'wordpress_jack'

switchboard = Switchboard::Client.new
switchboard.plug!(WordpressJack)

switchboard.on_new_post do |post|
  puts "A new post was received:"
  puts post
end

switchboard.on_new_comment do |comment|
  puts "A new comment was received:"
  puts comment
end

switchboard.run!

There are a number of improvements that can be made here (this is the “making blind assumptions about responses” version). Go wild.

Switchboard as a Framework

2009-07-19T00:00:00-07:00

Switchboard as a Framework

While Switchboard is a useful tool for debugging and probing XMPP services, it’s also a convenient and full-featured framework for building clients and components.

A Simple Client

This is the simplest client (bot) I could think of. It listens for input and replies with whatever was sent in the first place.

#!/usr/bin/env ruby -rubygems

require 'switchboard'

switchboard = Switchboard::Client.new
switchboard.plug!(AutoAcceptJack, EchoJack, NotifyJack)
switchboard.run!

Let’s break it down.

#!/usr/bin/env ruby -rubygems

require 'switchboard'

Um, I hope this is pretty straightforward.

switchboard = Switchboard::Client.new

This instantiates Switchboard::Client with some default options, mainly spin = true (this is the 2nd argument). spin means that #run! will cause the process to run in a loop and not return immediately. ^C will interrupt the process and shut it down cleanly. (^C a second time if it hangs.)

switchboard.plug!(AutoAcceptJack, EchoJack, NotifyJack)

This is the meat of it, even though it doesn’t look like it at first glance. Jacks are the basic units of shared functionality. More later, but the basic rundown is that AutoAcceptJack auto-accepts (and reciprocates) roster additions (“friend requests”), EchoJack does the heavy lifting of echoing input back, and NotifyJack sends presence (i.e. ”I’m online”) notifications to everyone (and everything) in your roster.

EchoJack (lib/switchboard/jacks/echo.rb) looks like this:

class EchoJack
  def self.connect(switchboard, settings)
    switchboard.on_message do |message|
      stream.send(message.answer)
    end
  end
end

I’ll get to the details of self.connect shortly, but the gist of it is that the EchoJack registers an on_message callback and uses Jabber::Message’s #answer method to create a response (inverting the sender and receiver) before sending it back on the stream (client or component connection, as the case may be).

Implementing the EchoBot as a jack is perhaps overkill, but the goal was to demonstrate how short and modular Switchboard apps can be. The alternate implementation looks like this:

switchboard.plug!(AutoAcceptJack, NotifyJack)
switchboard.on_message do |message|
  stream.send(message.answer)
end

Callbacks are executed in the context of the switchboard object; this is important to remember, as variables defined in a different scope will be unavailable.

switchboard.run!

This kicks off the process of connecting to the server. As XMPP is an asynchronous protocol, the different hooks will be called in response to varying inputs.

Jacks and Hooks

Jacks are extractions of standard functionality that are executed in the context of the Switchboard core. Thus, references to stream, etc. call methods implemented by Switchboard::Core and its subclasses rather than the jack itself. If you need convenience methods, they should be defined on the switchboard object provided as the 1st argument to connect:

def self.connect(switchboard, settings)
  def switchboard.helper_method
    # do something
  end
  
  switchboard.on_message do |message|
    helper_method
  end
end

connect is the entry point for all jacks. When the jack is plugged into switchboard (using Switchboard::Core#plug!), connect is called with the active Switchboard instance and its corresponding Switchboard::Settings. You can modify the settings in the body of connect, but what you’ll most often be doing is adding additional functionality (via method definitions or Modules) to the Switchboard instance.

PubSubJack (lib/switchboard/jacks/pubsub.rb) modifies the switchboard object by extending a helper Module:

def self.connect(switchoard, settings)
  switchboard.extend(Switchboard::Helpers::PubSubHelper)
  
  switchboard.on_startup do
    # ...
  end
end

Like everything else, jacks have access to all hooks (lifecycle callbacks). These are called using on_<hook name>. In general, these map to callbacks defined by xmpp4r.

startup
shutdown
stream_connected
exception
stanza
message
presence
iq

Clients provide the following additional hooks:

roster_presence
roster_query
roster_subscription
roster_subscription_request
roster_loaded
roster_update

Strictly speaking, these hooks should probably be implemented by a RosterJack and plugged in to Switchboard::Client by default.

Some jacks also introduce additional hooks (using hook(:name)), such as PubSubJack:

pubsub_event

This hook is registed in Switchboard::Helpers::PubSubHelper (lib/switchboard/helpers/pubsub.rb) with Switchboard::Core.hook(:pubsub_event).

Standard Jacks

The following jacks are available from the standard distribution:

AutoAcceptJack - Auto-accepts and reciprocates roster requests.
DebugJack - Displays color-coded <message />, <presence />, and <iq /> stanzas.
EchoJack - Example jack that echos inputs back to the sender.
NotifyJack - Sends online/offline presence to all roster items.
PubSubJack - Adds a pubsub_event hook and some PubSub convenience methods.
OAuthPubSubJack - Adds a pubsub_event hook and some OAuth-enabled PubSub convenience methods. The oauth gem must be installed for this to work.
RosterDebugJack - Like the DebugJack, but for roster events and colorless.

Fire Hydrant includes a FireEagleJack that introduces a location_update hook that yields a FireEagle::User object, demonstrating that Switchboard apps can be blissfully unaware of XMPP semantics when using the right jacks.

A More Complex Example, with Pandas

Bamboo Shooter is a pseudo-realtime interface to the Flickr Panda APIs. It polls the APIs once per minute and dribbles the responses out over XMPP over the course of the subsequent minute.

This represents a more complex example for several reasons. Firstly, it coexists with EventMachine as a set of additional threads (EventMachine is evented and thus single-threaded):

EM.run do
  Thread.new do
    # Bamboo::Shooter subclasses Switchboard::Component
    @shooter = Bamboo::Shooter.new(SETTINGS)
    @shooter.run!
  end
  
  # ...
end

Secondly, it implements a PubSub service as a component, so it subclasses Switchboard::Component rather than Switchboard::Client. A side-effect of implementing a service as a component is that the logic for everything that the server would ordinarily do has to be managed by your code instead. Presence handling (implemented by Bamboo::Shooter#presence_handler) is usually the first example of this.

PubSub requests and subscription handling are application-specific concerns, but much of the implementation is essentially boilerplate. Dovetail represents a start at abstracting this, but it’s incomplete at the moment. <subscribe /> and <unsubscribe /> requests are the only operations currently supported; the fallback is to return a <feature-not-implemented /> response:

def iq_handler(iq)
  if iq.pubsub
    if subscribe = iq.pubsub.first_element("subscribe")
      node = subscribe.attributes["node"]

      puts "Subscription to #{node} requested by #{iq.from}"
      subscribers[node] ||= []
      unless subscribers[node].include?(iq.from.strip)
        subscribers[node] << iq.from.strip 
      end

      resp = Jabber::Iq.new(:result, iq.from)
      resp.from = iq.to # TODO component.domain (elsewhere, too)
      resp.id = iq.id
      pubsub = Jabber::PubSub::IqPubSub.new
      subscription = Jabber::PubSub::Subscription.new(iq.from.strip, node)
      subscription.state = "subscribed"
      pubsub.add(subscription)
      resp.add(pubsub)

      deliver(resp)
    elsif unsubscribe = iq.pubsub.first_element("unsubscribe")
      node = unsubscribe.attributes["node"]

      puts "Unsubscription from #{node} requested by #{iq.from}"
      subscribers[node] ||= []
      subscribers[node].delete(iq.from.strip)

      resp = Jabber::Iq.new(:result, iq.from)
      resp.from = iq.to # TODO component.domain (elsewhere, too)
      resp.id = iq.id
      deliver(resp)
    else
      puts "Received a pubsub message"
      puts iq.to_s
      # TODO not-supported
      not_implemented(iq)
    end
  else
    # unrecognized iq
    not_implemented(iq)
  end
end

This is tightly tied to xmpp4r and REXML, but as the rest of Switchboard is too, it’s not a big deal for the time being.

Presence tracking is a somewhat tricky problem (particularly for large numbers of consumers), so you’ll notice that I completed punted and left it unimplemented. This means that subscriptions remain active (and will attempt to deliver photo payloads) even if a client has gone offline. For this reason, the consumers of this service are written to subscribe on startup and unsubscribe on shutdown. If the unsubscribe doesn’t happen, there’s a fair chance that the bamboo-shooter process’s memory usage will balloon. As a result, there is currently no public instance of this service running.

Polling of the Panda APIs is done using EventMachine:HttpRequest, which is a non-blocking, evented HTTP client. The result of each request (scheduled to run once per minute per panda) is parsed using REXML (because Switchboard is already using it via xmpp4r), split up, and scheduled for publishing using EventMachine.add_timer:

EM.run do
  # ...

  check_pandas = lambda do
    params = {
      'api_key' => SETTINGS["flickr.key"],
      'method' => 'flickr.panda.getPhotos'
    }

    ["ling ling", "hsing hsing", "wang wang"].each do |panda|
      req = EventMachine::HttpRequest.new('http://api.flickr.com/services/rest/')
      http = req.get(:query => params.merge('panda_name' => panda))

      http.callback do
        begin
          doc = REXML::Document.new(http.response)
          doc.root.each_element do |rsp|
            total = rsp.attributes["total"].to_s.to_f
            panda = rsp.attributes["panda"].to_s
            interval = rsp.attributes["interval"].to_s.to_f
            interval = interval / total
            delay = 0.0

            puts "#{panda} found #{total} items with a #{interval}s delay."

            rsp.each_element do |node|
              EventMachine::add_timer(delay) do
                @shooter.publish("/flickr/pandas/#{CGI.escape(panda)}", node)
              end
              delay += interval
            end
          end
        rescue REXML::ParseException
        end
      end
    end
  end

  EventMachine::add_periodic_timer(61, &check_pandas)
  check_pandas.call
end

Consuming the Bamboo Shooter feed can either be done by running switchboard pubsub --server <server> listen or with code like this (earth.rb, which will zoom Google Earth to the location where the photo was taken). Remember, Wang Wang is the Panda who likes maps.

#!/usr/bin/env ruby -rubygems

begin
  require 'appscript'
rescue LoadError => e
  gem = e.message.split("--").last.strip
  puts "The #{gem} gem is required."
end

require 'cgi'
require 'switchboard'

node = "/flickr/pandas/#{CGI.escape(ARGV[0])}"

earth = Appscript.app("Google Earth")

DEFAULTS = {
  "resource" => "earth",
}

settings = YAML.load(File.read("bamboo_shooter.yml"))
switchboard = Switchboard::Client.new(settings.merge(DEFAULTS))
switchboard.plug!(AutoAcceptJack, NotifyJack, PubSubJack)

switchboard.on_startup do
  defer :subscribed do
    puts "Subscribing to #{node}"
    subscribe_to(node)
  end
end

switchboard.on_shutdown do
  puts "Unsubscribing from #{node}"
  unsubscribe_from(node)
end

switchboard.on_pubsub_event do |event|
  event.payload.each do |payload|
    payload.elements.each do |item|
      photo = item.first_element("photo")
      lat = photo.attributes["latitude"].to_f
      lon = photo.attributes["longitude"].to_f
      earth.SetViewInfo({:latitude => lat,
                         :longitude => lon,
                         :distance => (rand * 25000) + 5000,
                         :azimuth => rand * 360,
                         :tilt => (rand * 75)},
                        {:speed => 1})
    end
  end
end

switchboard.run!

Publishing is managed by the Switchboard component:

def publish(node, xml_node)
  event = Jabber::PubSub::Event.new
  items = Jabber::PubSub::EventItems.new
  items.node = node
  item = Jabber::PubSub::EventItem.new

  item.add(xml_node)
  items.add(item)
  event.add(items)

  (subscribers[node] || []).each do |subscriber|
    message(subscriber, event)
  end
end

Getting Bamboo Shooter running is a little tricky, as it requires an XMPP that supports the component protocol (I use ejabberd with the following configuration). More complicatedly, this server either needs to be public (with DNS SRV records configured to support federation) or on a private network with a second XMPP server running. When developing locally, I use an Ubuntu VM with ejabberd running in component mode (using ejabberd.cfg below) and a second ejabberd instance with the default configuration running in OS X. bamboo-shooter.rb connects to Ubuntu, the client connects to OS X, and the ejabberd instances figure out how to connect to one another with ZeroConf (hostname.local; avahi-daemon on Ubuntu makes this possible).

{loglevel, 4}.
{hosts, ["localhost"]}.
{listen, [
{5222, ejabberd_c2s, [ {access, c2s}, {shaper, c2s_shaper}, {max_stanza_size, 65536} ]},
{5269, ejabberd_s2s_in, [ {shaper, s2s_shaper}, {max_stanza_size, 131072} ]},
{5288, ejabberd_service, [{host, "<hostname>", [{password, "<password>"}]}]}
 ]}.
{auth_method, internal}.
{shaper, normal, {maxrate, 1000}}.
{shaper, fast, {maxrate, 50000}}.
{acl, local, {user_regexp, ""}}.
{language, "en"}.
{modules, [
 ]}.

That’s it for now

That’s all I’ve got for now. It should be enough to get you started building clients and components, but if you have any questions, post a comment below or write to the Switchboard Google Group.

switchboard : XMPP :: curl : HTTP

2009-07-16T00:00:00-07:00

switchboard : XMPP :: curl : HTTP

Quick Start

Switchboard on GitHub: http://github.com/mojodna/switchboard/tree
Switchboard Google Group: http://groups.google.com/group/switchboard

Here’s how to install and use Switchboard for a few basic use-cases:

$ # install Switchboard
$ sudo gem install switchboard

$ # list everyone on your roster (buddy list)
$ switchboard --jid client@example.com --password pa55word \
    roster list

$ # add a friend to your roster
$ switchboard --jid client@example.com --password pa55word \
    roster add friend@example.com

$ # listen for PubSub events
$ switchboard --jid subscriber@example.com --password pa55word \
    pubsub --server <pubsub server> listen

“curl for XMPP?”

curl is the ultimate Swiss Army Knife for HTTP. Switchboard aims to be the same for XMPP.

HTTP is (relatively) easy. It’s stateless and has fairly limited semantics. However, if you’re trying to debug a web service and want to determine what certain ETags, additional headers, or specific content types respond with, curl is your go-to.

XMPP is a bit more complicated; it’s stateful, it (usually) requires login credentials, it’s asynchronous, and it has many extensions to the core protocol that are in varying levels of use. Traditionally, in order to explore an XMPP service, you’d have to delve into the advanced features of a client like Psi or Synapse, often dropping to the level of entering raw XML to see what happens in response.

Switchboard (and its command-line equivalent, switchboard) simplifies the process of repeatedly making common requests (e.g. roster manipulation, probing, and PubSub node operations) and is easily extended to support new behaviors.

Switchboard background

Copying and pasting XML isn’t the least error-prone process I can think of. In addition, I found myself repeatedly testing the same types of stanzas (e.g. PubSub requests) and attempting to implement extensions for which no implementations existed (i.e. OAuth over XMPP).

Diving into xmpp4r’s functionality led me to expose additional features to the command-line tool such as PEP and Last Activity to see how difficult it would be.

Since all of its dependencies have been released and are installable via RubyGems now, people who aren’t me can finally use Switchboard without much hassle. In honor of this milestone, Switchboard has been updated to 0.1.0. Real mature, I know, but it is still quite useful.

People who are me have been using this tool since last November (in fact, this is the project that inspired “My (public) Git Workflow”). (Some people have been using it to consume Fire Eagle’s Location Streams, but it hasn’t been for the faint of heart.)

Switchboard’s primary strength is certainly the command-line interface, but it turns out that it’s a pretty good abstraction over xmpp4r for building clients and servers (as components).

Here are a few projects I’ve worked on that use Switchboard:

bamboo-shooter - PubSub for Pandas: an pseudo-realtime XMPP interface to the Flickr Panda APIs
fire-hydrant - A simple library (as a Switchboard “jack”) for consuming Fire Eagle’s Location Streams
mars and jupiter - A matched pair of experiments mapping REST to XMPP
dovetail - A start at a framework for building PubSub servers (again, as components)

What to do?

Configuration

The first thing you’ll probably want to do with Switchboard is to provide some basic configuration so you won’t have to constantly enter your login credentials:

$ switchboard config jid jid@example.com
$ switchboard config password pa55word

To get the value of a setting, don’t include a value:

$ switchboard config jid # => jid@example.com

Some additional useful settings to set defaults for are:

debug - Turn on verbose mode: good for debugging. (true / false)
resource - Override the default resource of /switchboard (don’t include the /). This is useful if you want to run multiple instances with the same JID. This is equivalent to switchboard --resource <resource>. (String)
pubsub.server - Specify a default server to make PubSub requests against. This is equivalent to switchboard pubsub --server <server>. (String)
oauth - Use OAuth when making PubSub requests. This is equivalent to switchboard pubsub --oauth and requires that the oauth gem be installed. (true / false)
oauth.consumer_key - Specify a default OAuth consumer key. This is equivalent to switchboard pubsub --oauth-consumer-key <key>. (String)
oauth.consumer_secret - Specify a default OAuth consumer secret. This is equivalent to switchboard pubsub --oauth-consumer-secret <secret>. (String)
oauth.token - Specify a default OAuth token. This is equivalent to switchboard pubsub --oauth-token <token>. (String)
oauth.token_secret - Specify a default OAuth token secret. This is equivalent to switchboard pubsub --oauth-token-secret <secret>. (String)

In general, anything that is referred to in the library in the form OPTIONS["pubsub.node"] can be set as a Switchboard setting.

Roster Manipulation

Roster manipulation is something that can be easily handled by a desktop XMPP client, but sometimes it’s more convenient to be able to do it from the command-line.

Rosters can be listed, added to, or removed from. I’ll assume you’ve configured Switchboard with some login credentials.

$ switchboard roster list
$ switchboard roster online
$ switchboard roster add friend1@example.org friend2@example.org
$ switchboard roster remove friend2@example.org enemy@example.org

Probing and Discovery

XMPP provides some pretty heady functionality when it comes to determining what a server is capable of. A disco#info query is the first step to use when determining capabilities:

$ switchboard disco --target jabber.org info

The response to this query includes http://jabber.org/protocol/disco#items, which means that jabber.org supports disco#items queries, which allow you to determine what top-level items (services) are available:

$ switchboard disco --target jabber.org items

This response includes conference.jabber.org. Let’s list items available there:

$ switchboard disco --target conference.jabber.org items

Whoa. A list of MUCs (multi-user chats) hosted on conference.jabber.org.

You can do the same thing to list available PubSub nodes if you’re running a local copy of ejabberd or another XMPP server that supports it. (Note that you may need to prefix your hostname with pubsub in order to see the nodes.)

PubSub

Switchboard supports more of PubSub than any other extension, mainly because that’s been my primary focus of XMPP experimentation. To get a full list of available PubSub commands:

$ switchboard pubsub

A basic sequence of events is to subscribe:

$ switchboard pubsub --server <server> subscribe --node <node>

List subscriptions:

$ switchboard pubsub --server <server> subscriptions

Listen for notifications:

$ switchboard pubsub --server <server> listen

Unsubscribe:

$ switchboard pubsub --server <server> unsubscribe --node <node>

Let’s walk through a couple examples.

First, Superfeedr, which bills itself as “real-time feed parsing in the cloud”. To begin, you’ll need to register and activate your account. Once you’ve done that, set up a subscription a Twitter search for “xmpp”:

$ switchboard --jid <username>@superfeedr.com --password <password> \
    pubsub --server firehoser.superfeedr.com \
    subscribe --node "http://search.twitter.com/search.atom?q=xmpp"

We would next list subscriptions, but Superfeedr uses a non-standard mechanism to do so. Instead, let’s listen for new results:

$ switchboard --jid <username>@superfeedr.com --password <password> \
    pubsub --server firehoser.superfeedr.com listen

If you’re lucky, you’ll get an Atom payload or two. Here’s one:

<event xmlns='http://jabber.org/protocol/pubsub#event'>
  <status feed='http://search.twitter.com/search.atom?q=xmpp' xmlns='http://superfeedr.com/xmpp-pubsub-ext'>
    <http code='200'>16933 bytes fetched in 0.600034s</http>
    <next_fetch>2009-07-22T17:26:22Z</next_fetch>
  </status>
  <items node='http://search.twitter.com/search.atom?q=xmpp'>
    <item chunk='1' chunks='1'>
      <entry xmlns='http://www.w3.org/2005/Atom'>
        <title>Great... trillian update has killed my ability to view my xmpp rosters</title>
        <summary>Great... trillian update has killed my ability to view my &lt;b&gt;xmpp&lt;/b&gt; rosters</summary>
        <link href='http://superfeedr.com/entries/tr5gfgstf8oqlcgr5opaeotxk39ovtos0oiat7h12mqfuoxdmgbjz1rnjtzswqvja2dqh8cgg31' rel='alternate' type='text/html'/>
        <published>2009-07-22T17:10:52Z</published>
        <id>tag:search.twitter.com,2005:2781212809</id>
      </entry>
    </item>
    <item chunk='1' chunks='1'>
      <entry xmlns='http://www.w3.org/2005/Atom'>
        <title>usando Tkabber: TKabber is Tcl/Tk Jabber-client with great functionality. It supports MUC, XMPP-statuses a.. http://bit.ly/AsFPg</title>
        <summary>usando Tkabber: TKabber is Tcl/Tk Jabber-client with great functionality. It supports MUC, &lt;b&gt;XMPP&lt;/b&gt;-statuses a.. &lt;a href=&quot;http://bit.ly/AsFPg&quot;&gt;http://bit.ly/AsFPg&lt;/a&gt;</summary>
        <link href='http://superfeedr.com/entries/w6cezbjniqqgga3bd79ruklxhu7f4a6qqpro76hgz50gzccuekgmehz39yb1zi1cclgo83s' rel='alternate' type='text/html'/>
        <published>2009-07-22T17:07:54Z</published>
        <id>tag:search.twitter.com,2005:2781162048</id>
      </entry>
    </item>
  </items>
</event>

It includes a Superfeedr-specific <status/> element with information on the most recent fetch as well as standard Atom feeds contained within standard <item/> elements. This means that you can extract the Atom elements from the PubSub payload and hand it to a feedparser of some variety to work its magic.

Fire Eagle’s Location Streams also use PubSub, but subscription management requires that requests are signed using OAuth. This allows 3rd party applications to make requests on behalf of specific users without having to obtain their actual credentials (in short, it’s exactly the same use-case for OAuth over HTTP).

Let’s start with a subscriptions list request, since we have the credentials (“General Purpose Access Token”) immediately after registering a “web” application with Fire Eagle.

$ switchboard pubsub --oauth \
    --oauth-consumer-key <consumer key> \
    --oauth-consumer-secret <consumer secret> \
    --oauth-token <general token> \
    --oauth-token-secret <general token secret> \
    --server fireeagle.com \
    subscriptions

Odds are, you’ll have nothing there. Let’s change that. Send yourself through the authorization process in order to get a valid OAuth token and secret:

$ oauth --consumer-key <consumer key> \
    --consumer-secret <consumer secret> \
    --access-token-url https://fireeagle.yahooapis.com/oauth/access_token
    --authorize-url https://fireeagle.yahoo.net/oauth/authorize
    --request-token-url https://fireeagle.yahooapis.com/oauth/request_token
    authorize

With that token and secret, subscribe to your Location Stream:

$ switchboard pubsub --oauth \
    --oauth-consumer-key <consumer key> \
    --oauth-consumer-secret <consumer secret> \
    --oauth-token <token> \
    --oauth-token-secret <token secret> \
    --server fireeagle.com \
    subscribe --node "/api/0.1/user/<token>"

Now you’ll have a subscription to query for:

$ switchboard pubsub --oauth \
    --oauth-consumer-key <consumer key> \
    --oauth-consumer-secret <consumer secret> \
    --oauth-token <general token> \
    --oauth-token-secret <general token secret> \
    --server fireeagle.com \
    subscriptions

Listen for location updates:

$ switchboard pubsub --server fireeagle.com listen

Update your current location and watch as the update rolls in. If you’d like to visualize updates with Google Earth, check out fire-hydrant on GitHub.

We’re done, so we may as well clean up and unsubscribe:

$ switchboard pubsub --oauth \
    --oauth-consumer-key <consumer key> \
    --oauth-consumer-secret <consumer secret> \
    --oauth-token <token> \
    --oauth-token-secret <token secret> \
    --server fireeagle.com \
    unsubscribe --node "/api/0.1/user/<token>"

PEP (Personal Eventing Protocol)

PEP is a specialized version of PubSub, intended to allow individuals to associate data with their JIDs. Switchboard supports publishing of User Tune and User Location.

Due to the ability of XMPP to allow multiple instances of the same account online (identified with different resources), Switchboard can serve up User Tune and User Location for the same account you’re already online with. Not every client supports displaying them, but they’re fun to play with regardless.

To publish User Tune, you need to be on a Mac, running iTunes, and have the rb-appscript gem installed (sudo gem install rb-appscript). Once that’s done:

$ switchboard --resource switchtunes pep tune

To publish User Location, you need to be updating Fire Eagle (Clarke is an excellent background updater for OS X) and have the fire-hydrant gem installed from GitHub (sudo gem install mojodna-fire-hydrant -s http://gems.github.com). Once that’s square:

$ switchboard --resource switchfire pep location

Switchboard supports more functionality than I’ve described above. To get a list of general switchboard commands (some of which may have sub-commands):

$ switchboard

Getting Help

In theory, if you want more information about a specific command, you can use switchboard help <command>. For example:

$ switchboard help pubsub

For now, you’ll notice that it’s not particularly useful. If you’d like to help rectify this, you can implement various help methods that are lying around, such as Switchboard::Commands::PubSub.help.

Extending

Writing new Switchboard commands is really easy, assuming that the primary application logic that you’re depending exists elsewhere (i.e. in xmpp4r).

I was on a panel with Peter St. Andre and Jack Moffitt at the Glue Conference in Denver this Spring and we got to talking about tools like Switchboard. Jack wondered how hard it would be to implement something like grep for XMPP.

(Jack is one of the authors of a Python project similar to Switchboard named poetry).

I took a whack at it and it came out like this:

module Switchboard
  module Commands
    class Grep < Switchboard::Command
      description "Search for an XPath expression"

      def self.run!
        expr = ARGV.pop

        switchboard = Switchboard::Client.new
        switchboard.plug!(AutoAcceptJack, NotifyJack)

        switchboard.on_stanza do |stanza|
          # TODO doesn't handle default namespaces properly
          REXML::XPath.each(stanza, expr) do |el|
            puts el.to_s
          end
        end

        switchboard.run!
      end
    end
  end
end

Jacks (not Moffitt) deserve their own discussion, but the thrust of this piece of code is the #on_stanza callback (which yields a REXML::Node object) and the XPath expression. Note that for whatever reason, you can’t search for nodes that have a default namespace (e.g. <presence />); I’m going to assume that this is a REXML quirk.

If you want to take a shot at implementing a Switchboard command, Ping should be pretty simple. Alternately, Switchboard doesn’t support sending or receiving basic <message /> stanzas from the command-line. Supporting those would make it simple to interact with a service like identi.ca.

How to Help

There are a few simple ways to help out with Switchboard’s development:

explore the API
build out the docs
implement IM functionality
write up how to use it for a particular service / use-case
use it!

What’s your favorite use for Switchboard?

Updating Ruby Consumers and Providers to OAuth 1.0a

2009-05-20T00:00:00-07:00

Updating Ruby Consumers and Providers to OAuth 1.0a

In a previous post, I did a quick run-through of the changes that were introduced in OAuth 1.0a. As promised, here’s a rough guide to updating Ruby Consumers and Providers to 1.0a. Don’t mind the pseudo code.

Updating Ruby OAuth Consumers to 1.0a

In order for things to work properly, you’ll need to use a version of the OAuth gem that’s at least 0.3.4.1 (0.3.5 was released on 6/3/09). To install it and check the version number:

$ sudo gem install oauth
$ oauth --version

Authorization code that once looked like this:

request_token = consumer.get_request_token
puts "Please visit the following URL to authorize this application:"
puts request_token.authorize_url(:oauth_callback => callback_url)
# wait for input
gets
access_token = request_token.get_access_token

Should now look like this:

request_token = consumer.get_request_token(:oauth_callback => callback_url)
puts "Please visit the following URL to authorize this application:"
puts request_token.authorize_url
# wait for input
puts "What's the value of `oauth_verifier`?"
oauth_verifier = gets.chomp
# `oauth_verifier` is extracted from the expanded callback URL or was displayed to the user
access_token = request_token.get_access_token(:oauth_verifier => oauth_verifier)

You can detect whether a Service Provider supports 1.0a:

request_token = consumer.get_request_token
puts "OAuth 1.0a detected" if request_token.callback_confirmed?

Updating Ruby OAuth Providers to 1.0a

As with the updates to consumer applications, you’ll need to be running at least 0.3.4.1. For the sake of simplicity (and additional laziness on my part), I’ll assume OAuth verification is baked into your Rails app rather than as Rack middleware (where it probably belongs).

You’ll need to add a pair of columns to your oauth_request_tokens table (or whatever it’s called): callback and verifier.

Accepting `oauth_callback` During the Request Token Phase

The first step to supporting OAuth 1.0a is to accept oauth_token parameters when issuing Request Tokens. To do this, you’ll need to make the OAuth::RequestProxy::ActionControllerRequest available to methods that run later in a request’s lifecycle:

def verify_oauth_signature
  valid = OAuth::Signature.verify(request) do |request_proxy|
    # make the request proxy available outside this block
    @_request_proxy = request_proxy
    
    # proceed normally...
  end
end

# accessor for the request proxy
def oauth_request_proxy
  @_request_proxy
end

Once this is set up, you’ll need to modify your request_token method to associate the oauth_callback parameter with your Request token and set oauth_callback_confirmed to true:

def request_token
  request_token = new_request_token
   # request_proxy provides unified interface to params + headers
  request_token.callback = request_proxy.oauth_callback
  request_token.save
  
  render :text => "oauth_token=#{request_token.token}&" \
                  "oauth_token_secret=#{request_token.secret}&" \
                  "oauth_callback_confirmed=true"
end

Generating An `oauth_verifier` During the Authorization Phase

In addition to validating the Request Token during the authorization phase, you’ll want to generate an oauth_verifier value and return it to the application via the pre-registered callback url (or display it to the user and instruct them to enter it into their application).

def authorize
  # display the authorization page
  
  render and return unless request.post?

  # validate the token

  request_token = find_request_token(params[:oauth_token])
  request_token.validated = true
  # generate a verification code
  request_token.verifier = generate_verifier
  request_token.save
  
  if request_token.callback?
    # this was previously params[:oauth_callback]
    redirect_to request_token.callback + "?oauth_verifier=#{request_token.verifier}"
  else
    @verifier = request_token.verifier
    render # display the verification code to the user
  end
end

Verifying Access Token Exchanges

When exchanging a Request Token for an Access Token, you need to confirm that the verification code provided by the consumer matches the one on file.

def access_token
  # this is a correctly signed request: oauth_token has already been loaded

  # compare the verifier from the request proxy to the one we generated
  if oauth_token.verifier == request_proxy.oauth_verifier
    # exchange the request token for an access token
    access_token = oauth_token.exchange
    render :text => "oauth_token=#{access_token.token}&" \
                    "oauth_token_secret=#{access_token.secret}"
  else
    raise OAuth::InvalidVerifier
  end
end

Summary

Obviously, there are cleaner ways to do this, but they’re presumably very specific to individual codebases. If you have questions, check out the oauth-ruby mailing list. Otherwise, patches can be submitted against http://github.com/mojodna/oauth/tree/mergeable.

Good luck!

An Idiot's Guide to OAuth 1.0a

2009-05-20T00:00:00-07:00

An Idiot’s Guide to OAuth 1.0a

I’m lazy. I don’t usually enjoy re-reading things I’ve already read before (Matt Ruff and Neil Stephenson books are an exception). So, as a service to all 3 of my readers, I’ll summarize the changes to the OAuth specification for 1.0a (draft 3) as I understand them. Finding changes in a large document is a pain–you have better things to do with your time. Embrace your inner ignorance and leave responsibility to the wind.

As an added bonus, I’ll demonstrate how to update consumer and provider code using the Ruby library.

What Happened?

Much has been written about this elsewhere, so I’ll be brief. A session fixation attack was discovered a little over a month ago. Essentially, it affixes your sessions to someone else’s. Imagine burrs or inverted balls of duct tape with little teeny messages (on grains of rice) attached to them.

Anyway. It was bad. It was one of those things that once you’ve seen, it’s impossible to un-see. There was no straightforward fix that could be deployed, so you started seeing lots of BIG ANGRY warning notices explaining that your car might be repossessed if you were bold enough to continue.

The longer-term solution was to change the specification, get providers on-board to change their code, and finally to reach out to developers to update their client applications. All so the world doesn’t implode next time you turn right. DON’T TURN RIGHT. Ok?

So, What Changed?

Callbacks (to your site or application) have been supported in various forms (with varying restrictions), but they have always been parameters passed to the Service Provider’s authorization page. Instead, the oauth_callback parameter is now part of the Request Token phase. If your client cannot accept callbacks, the value MUST be oob. SPs use this to determine whether a client supports 1.0a.
oauth_callback_confirmed will be present (indeed, MUST according to the spec) when Service Providers supporting 1.0a issue a Request Token. (e.g. oauth_token=asdf&oauth_token_secret=qwerty&oauth_callback_confirmed=true) Clients can use this to determine whether a server supports 1.0a.
An oauth_verifier parameter is provided to the client either in the pre-configured callback URL or through the fingers of your users (the aforementioned oob (“out of band”) mechanism). This value MUST be included when exchanging Request Tokens for Access Tokens.

The changes to the spec are limited to the Request Token ➡ Access Token exchange, so once you have an Access Token, everything should behave as it did before. For this reason, clients that only implement 2-legged OAuth are unaffected.

Section 11.14. Cross-Site Request Forgery (CSRF) is also worth reading (and understanding) in order to further secure the use of callback URLs.

Services Supporting OAuth 1.0a

At the time of this writing, the only service provider I’m aware of that supports 1.0a is Yahoo!. This doesn’t include Fire Eagle (unless you include my laptop), although we intend for this to change in the next couple weeks. Google’s OAuth endpoint should support 1.0a soon, but I’ve yet to see any confirmation.

My (public) Git Workflow

2009-03-09T00:00:00-07:00

My (public) Git Workflow

The problem: You want to be an effective contributor to a public project that’s using Git.

The solution: Follow conventions and keep clean branches to make them easier to merge.

Let’s Go

For your own projects, it doesn’t really matter what you do to publish and maintain your code, at least until other people start forking and attempting to hack on your stuff. In any case, you should keep things simple. Use topic branches for particular features or refactoring if you want, but no big deal.

Determining “Active” and “Authoritative” Forks

GitHub’s search attempts to surface active projects by displaying the number of forks, number of watchers, and recent activity, but it doesn’t tell the full story. The “root” of the fork tree (which isn’t really a tree, but is kinda useful to think of that way) isn’t necessarily the current focus of activity. Someone may have published a project and lost interest, but a community may have formed around one of the forks.

Alternately, a project may have been imported from Subversion by a wannabe contributor prior to the original author joining GitHub and applying community-created patches to his or her own fork (which, at this point, isn’t the root of the fork tree).

If there are a relatively low number of forks (say, one or two), you can (mostly) assume that the fork with the most activity is authoritative. Other contributors, with forks that come and go or that are generally out of date, can be considered “outsiders.”

For projects with a larger number of forks, there may be multiple authorities (and potentially even multiple development directions), with many outsiders periodically contributing patches.

All of this means that you, as a potential contributor, need to do some additional research to determine which fork (or forks) is “active” and/or “authoritative”. They may not be one and the same; the “authoritative” fork may focus on stability; you may need bleeding edge features developed by an “authoritative” member of the community instead.

GitHub’s Network Graph is a great tool for this purpose. From the interface, you can see which forks have been the most active (hint: more •’s), most recently active (•’s further right), or have had the most merges (more ⤴’s). Those don’t tell the full story, so you’ll want to mouse-over individual •’s to see who wrote the patch (and who committed it). That will help distinguish between the forks where the patches originated and where they’ve merely been pulled in. You’ll also get a sense of the number and names of people working on a project.

Some Rules of Thumb

When hacking on other peoples’ stuff, your eventual goal is (probably) to get those changes applied. Here are a few tips to make life easier for everyone involved.

First rule of thumb: merge from authoritative contributors, cherry pick from outsiders.

By merging (e.g. git merge user/master) from authoritative contributors, it becomes easier for them to apply your changes if and when they deem them fit. Merging from outsiders may force the authorities into accepting patches (or series) that they may not be ready for (as those branches may include changes unrelated to what you were working on).

GitHub’s Fork Queue cherry picks (e.g. git cherry-pick 469df8) commits by default, so that’s a good way to get your tree in order before adding your own changes. If there are changes to your fork’s parent, you can fast forward your fork before starting work.

Second rule of thumb: keep your changes easily apply-able to the authoritative repository.

For a single line of development, this is easy: rebase against the authoritative master branch whenever possible. Feel free to rewrite history (i.e. git push -f) as long as you’re confident no one is treating you as authoritative.

If others are treating you as an authoritative source, create a separate branch that gets periodically rebased and make it clear that it’s not suitable for tracking. Others’ changes will eventually need to be rebased against those changes, but probably not until your changes have been applied by a real authoritative contributor.

For changes that do many things, this means keeping multiple branches active. I usually use a branch per feature, plus a separate branch for general bugfixes and documentation. For minor changes, all branches can diverge from a common commit (usually the tip of the authoritative repository’s master branch).

For more drastic changes, you’ll need to decide which is either the most important feature or the least controversial change. Use that as the first patch series and rebase it against the authoritative repository’s HEAD. Rebase subsequent patch series against the next most important (or least controversial).

Write down the order in which your branches should be merged. You can use your project wiki or put it at the top of the README in your master branch. (See my fork of xmpp4r to see how I’ve done it.) At this point, your master branch is just the default view of your fork and shouldn’t be considerably different from the branch of the authoritative copy you’re tracking (see the exception below).

In some cases, you’ll have branches that aren’t quite ready for prime-time. Treat them the same as branches that are, but don’t make apply-able branches dependent on them (i.e. they should always be the last changes to be applied). Include them in the summary in your README, but make it clear that they’re not ready yet.

Third rule of thumb: reset your history to match an authoritative repository’s history if their content is the same (or essentially the same).

If your history frequently diverges from an authoritative repository’s history, you’re going to have more trouble applying others’ changes and synchronizing with authoritative repositories.

The simplest way is to delete your repository and re-fork. If you have minor changes, generate a diff to apply once you’ve re-forked.

An Exception

Because GitHub will only build gems for changes on the master branch, if you have changes that you want to publish (prepared neatly in topic branches, as above), you’ll need to break the rules somewhat.

To handle this scenario, start with your master branch matching the authoritative branch you’re tracking, then merge branches like the owner of the authoritative branch would (this has the helpful side-effect of discovering whether your changes will apply cleanly).

The tip of your master should always contain the most up-to-date gemspec (in order to trigger GitHub’s building and publishing of your fork) as well as a README that contains information on the state of your repository (including branches, as above, but also information about how you’re treating the master branch).

Maintaining a Project

Over the last couple months, I’ve become one of the authoritative forks for the OAuth gem, indirectly resulting in this document. I’ve attempted to be consistent in the management of my fork, including publishing periodic releases for others to test functionality, but it became clear that my methods weren’t as transparent as they ought to be.

As a maintainer (or authoritative fork), you frequently find the need to merge in work from other developers. GitHub’s Fork Queue is excellent for this, but it’s not the last word.

The Fork Queue is perfect for merging individual commits, but when someone submits a patch series, it’s time to make a decision: you can cherry-pick (e.g. git cherry-pick 469df8) all of their commits (which is what the Fork Queue does behind the scenes)–retaining authorship but setting the committer field to yourself–or you can pull their branch down and perform a merge (e.g. git merge user/master), retaining both author and committer.

Cherry picking commits is easier–unless you want/need to make adjustments to them (in which case the SHAs would change anyway)–and results in linear history (no merges) with the downside of having repetitive commits (with different committers) across the project network, thus confusing GitHub’s Network Graph (or anything that displays the history of multiple branches) and making it more difficult to find un-applied patches (as the same patch may exist with multiple SHAs).

Merging changes locally is more difficult, but the github gem makes it easier by pulling in the entire patch universe with gh network fetch. Once that’s been done, the combination of gh network commits, GitX, and the Network Graph make it relatively easy to identify commits ripe for picking.

An advantage of cherry picking is that you have the freedom to amend (git commit --amend) commits before pushing them out as part of your tree. This is a great opportunity to clean things up to match local coding standards (that should already be the case, but YMMV) and to add additional context to the commit message.

Note: You can achieve the same effect as amending commits by using git format-patch -1 469df8 to create a patch, applying it with git apply, pausing to clean it up, and committing it with git commit -c 469df8. git ready has a good article on picking out individual commits.

The main problem with cherry picking becomes more acute when there are multiple authoritative forks (which should maintain a consistent public state) constantly passing work back and forth. If the participants periodically merge others’ work, you end up with a number of merge commits, but a relatively coherent history. If one or more participants choose to cherry pick commits instead, merging their branches back will include repeated commits with different SHAs and committer fields. The resultant state of the tree is fine, but the history gets somewhat polluted and becomes more difficult to follow.

This can be mitigated by sending patches via email (or any mechanism by which public repositories don’t change) and repeatedly munging history locally, but that’s often not an option, given the level of coordination required.

Aside: Complexity and / or Maturity

I used to see complicated Network Graphs (with lots of merging and branching) as a sign of project activity, but now I consider it more as a measure of contributors’ comfort with git (as individual contributors may be responsible for multiple branches). I believe that the main reason that projects end up with complicated history is due to (relatively) limited coordination between collaborator and maintainer; if I had to guess, I would expect closed projects using Git to have much more linear histories.

Ultimately, I think complicated histories are a side-effect of Git’s merging prowess promoting and supporting the drive-by contributor model. However, as powerful as Git is, coordination between collaborators still lessens the burden on the person doing the merging, thus the above manifesto.

Now What?

Get yourself out there, coding whatever and whenever, and contribute back. You’ll feel good and you will have made the world a little better, one patch at a time.

My (work) Git Workflow

2009-02-24T00:00:00-08:00

My (work) Git Workflow

The problem: You want to track multiple patchsets against an upstream Subversion repository easily.

The solution: Use git-svn to track it and create topic branches for local changes.

Jump here for my actual workflow

A Not-So-Brief Introduction (with code!)

I started using git and git-svn a little over a year ago. I was an Amtrak regular at the time, so the ability to make offline commits seemed helpful. Plus, it was shiny, and I’d seen a demo of GitHub, so I had a hunch that it was going to be big.

One day, I sat down over lunch and watched most of Linus Torvalds’ git talk at Google. Until then, I hadn’t really considered DVCSes, staying in my comfortable little Subversion world, branching out to cvs and bzr as necessary. I don’t really remember what I got out of the talk, but somehow it inspired me to start reading up on how git could be used with svn, much like svk.

git-svn made its debut performance on our team’s IRC channel about a month later; I had been doing some major refactoring on a local git branch with lots of commits (squashing commits was mystery to be discovered later) and the time had come to merge it into trunk. The merge went fine (lucky, I guess), so I went for a git svn dcommit. And spammed the IRC channel for the next 30 seconds or so.

Flash forward to the following winter and new (remote) contributors to our project. It was time to start doing legitimate code reviews, in part to get the new folks up to speed with the code-base.

In the process of setting up Review Board, I discovered that it doesn’t play well with diffs generated with git. Bummer. They’re nice. I spent some time poking around RB’s source tree to see why that was the case. Long story short, no dice. However, on one of the message boards where the problem had been raised (no resolution, natch), I saw a snippet to transform git diffs into svn diffs, revision numbers and all. Bingo. After fiddling with it for a while, I ended up with the following, which has been working well.

Add the following to your ~/.gitconfig to enable git svn-diff. Awesome. It even looks like it belongs there. (git-svn-diff must be in your PATH.)

# ~/.gitconfig
[alias]
  svn-diff = !git-svn-diff

add git svn-diff as an alias for git-svn-diff

Bonus Points

Here are the relevant pieces of my ~/.bashrc to get tab completion and a colorized prompt that includes git branch information in it:

The Sizzle

This is essentially the git-svn variation of Josh Susser’s pure git workflow.

The Setup

Clone the target Subversion repository:

$ git svn clone svn+ssh://svn.host/path/to/repo -s

assuming a standard Subversion layout

Alternately, if you’re planning on sharing topic branches with fellow developers, have one person do the clone and pass it out (by copying it, to retain Subversion metadata).

Fixing a Bug / Implementing a Feature

Make sure you’re up-to-date:

[master] $ git svn rebase

Create a topic branch (I include a title to recognize it more easily) and check it out:

[master] $ git checkout -b bug-42-title

Attempt a bug fix (write a test, make it pass):

[bug-42-title] $ ...

Check it in:

[bug-42-title] $ git commit -a

Generate a patch for review:

[bug-42-title] $ git svn-diff > bug-42-title.patch

this will diff against the checked out trunk revision

Post it for review.

Time passes… Your diff hasn’t been reviewed, development continues…

Update the tracking branch:

[master] $ git svn rebase

Rebase your topic branch against the current trunk:

[master] $ git checkout bug-42-title
[bug-42-title] $ git rebase master
[bug-42-title] $ # resolve conflicts; `git mergetool` is handy

Regenerate the patch:

[bug-42-title] $ git svn-diff > bug-42-title-2.patch

Post it for review.

Time passes… Your diff has been reviewed and been found wanting…

Make changes to your topic branch:

[bug-42-title] $ ...

Check them in:

[bug-42-title] $ git commit -a

Regenerate the patch:

[bug-42-title] $ git svn-diff > bug-42-title-3.patch

Post it for review.

Landing a Reviewed Patch

You have a few options here.

You can apply the patch directly to the tracking branch:

[master] $ git apply bug-42-title-3.patch
[master] $ git commit -a

If you want to preserve history (i.e. multiple commits that tell a story), update the tracking branch and rebase your topic branch against it before merging:

[master] $ git svn rebase
[master] $ git checkout bug-42-title
[bug-42-title] $ git rebase master
[bug-42-title] $ # resolve conflicts
[bug-42-title] $ git checkout master
[master] $ git merge bug-42-title

If you want to get fancy (and remove your frustrated profanity), do an interactive rebase on the topic branch before merging:

[bug-42-title] $ git rebase -i
[bug-42-title] $ git checkout master
[master] $ git merge bug-42-title

Whew. Almost done. You’ll want to update the upstream Subversion repository, lest you risk your hard work being wasted:

[master] $ git svn dcommit

this will git svn rebase if necessary

If you don’t have write access to the upstream repository, submit the patch by mail instead.

Finally, once the patch has been merged, you can clean up your local repository by removing the topic branch:

[master] $ git branch -d bug-42-title

Topic branches needn’t to be limited to your own bug-fixes. git apply is quite liberal in what it understands, so you can grab diffs (from ReviewBoard, for example) and apply them to new branches to try out uncommitted functionality (or as an aid to the code reviewing process).

That’s it! You can keep as many topic branches active as you want (or need to), rebasing them against your tracking branch as necessary. I use this approach for bugs, features, and exploratory refactoring that may never see the light of day.

Working with Subversion Branches

Working with Subversion branches is remarkably straightforward, but it’s necessary to understand a few things about the repository layout and the branches that git-svn sets up when it does the initial clone.

First, if you did a git svn clone -s against a Subversion repository with a typical layout (trunk, branches/<branch>, tags/<tag>), a number of remote Git branches will have been created: one for trunk, one for each branch, and one for each tag (odd, but sensible given that Subversion doesn’t strictly do tags).

git branch will list local branches:

[master] $ git branch
  bug-42-title
* master

git branch -r will list Subversion branches (and tags):

[master] $ git branch -r
  stable-1.0
  tags/REL_1.0
  trunk

By default, your local master branch will be set up to track the remote trunk branch. Instead of using git pull (in a pure Git workflow) to update trunk to the current remote revision, use git svn rebase.

If you want to ensure that you have up-to-date versions of all Subversion branches, use git svn fetch to load upstream commits. This has the side-effect of causing master and trunk to be out-of-sync, so you’ll have to follow that with git svn rebase from the master branch. GitX will show you the current state of both local and remote branches, and thus is immensely useful when attempting to determine your current repository state.

In case you didn’t fully grok that last paragraph, where you would have used git svn rebase without branches, you should do the following (rebasing still works, but your branches won’t be up-to-date):

[master] $ git svn fetch
[master] $ git svn rebase

git svn rebase in this case is the equivalent to git rebase trunk, as trunk is the “SVN parent of the current HEAD” (from git help svn).

git svn dcommit will use svn to commit outstanding changes upstream (again, to the “SVN parent of the current HEAD”). The upstream branch in question depends on the Subversion branch that your current branch is tracking, usually using git branch --track <local> <remote>.

Subversion branches and “tags” can also be created with git-svn. For example, to tag and branch a 1.1 release:

[master] $ git svn branch -m "branching for 1.1" stable-1.1
[master] $ git svn branch -m "1.1 release" -t REL_1.1
[master] $ git branch -r
  stable-1.0
  stable-1.1
  tags/REL_1.0
  tags/REL_1.1
  trunk

Commit messages are necessary, as branching in Subversion is the equivalent to creating a changeset where a directory is copied.

Now that the branch has been created upstream, create a local tracking branch such that git svn dcommit will commit to the correct upstream branch.

[master] $ git branch --track stable-1.1 stable-1.1
[master] $ git checkout stable-1.1
[stable-1.1] $

Extra Credit

Convert Subversion “tags” into proper Git tags:

#!/bin/sh
#
# git-svn-convert-tags
# Convert Subversion "tags" into Git tags
for tag in `git branch -r | grep "  tags/" | sed 's/  tags\///'`; do
  git tag $tag refs/remotes/tags/$tag
done

Add the following to your ~/.gitconfig to enable git svn-convert-tags. (git-svn-convert-tags must be in your PATH.)

# ~/.gitconfig
[alias]
  svn-convert-tags = !git-svn-convert-tags

Extending ActiveRecord Attributes

2007-06-14T00:00:00-07:00

Extending ActiveRecord Attributes

David Black came to speak at the Boston Ruby Group on Tuesday (along with Zed Shaw, making it the most star-studded event since I started going). It was in one of the lecture halls at the Harvard Science Center (which, if you follow the pattern of every single other building on campus, is named after the “Science Center” family; seems like it should be Merrill, or Morrill, or something instead–here’s your chance). I was expecting stand-up comedy, which turned out to be an unreasonable expectation because it was in lecture hall A, not C (which is where HSUCS (a group of stand-up comedians at Harvard) has had events in the past). But I digress.

David’s talk was about “Per Object Behavior in Ruby” and about treating it as a prototype language (a logical follow-up to Dave Thomas’ RailsConf keynote; he didn’t disappoint and provided classless code (except for the tests; that would have been a nice touch)). I won’t go much into the substance of the talk, except to say that he managed to clarify the meaning and use of an object’s “singleton class” / “eigenclass” as well as further emphasizing that class != type, but that a class is merely a template from which new objects spring. He encouraged people to take advantage of Ruby’s ability to extend individual instances of objects with methods (generally via modules), as that’s a considerable part of what makes Ruby Ruby.

I was reminded that I had written some ActiveRecord code this spring that allows “primitive” (String, Fixnum) attributes of AR objects to be embellished by extending the attributes’ objects before returning them from the containing object.

In our case, we had a notion of FormattedText objects, which were essentially Strings that could contain HTML. Unfiltered HTML, as it turned out. Rather than filtering during the input stage, we decided that we would store what the user provided and leave it to the output stage to handle the filtering. Instead of using a helper method in each and every case (with the downside of having to remember to use it), I decided that the filtering functionality would fit better within the object (so it would be used appropriately in the majority case without any extra thought). In this case, by overriding String#to_s.

module FormattedText
  def to_s
    # filter and output
  end
end

The next step was making sure that only attributes known to contain formatted text were extended by FormattedText (and that the internal representation remained untouched, so that the filtered version wouldn’t be saved back to the database). A side-effect of this is that changing the value that was returned and saving the object will not save the new value back to the database; to get around this, you’ll need to set the value on the AR object itself.

# Quick and dirty monkey-patch (should work with 1.1.x+)
class ActiveRecord::Base
  # Attribute extension
  cattr_accessor :extended_attributes
  @@extended_attributes = {}

  class << self
    # create the "extends" declaration
    def extends(attr_name, options)
      extended_attributes[attr_name.to_sym] = options[:with]
    end
  end

  alias_method :define_read_method_without_extends, :define_read_method
  def define_read_method_with_extends(symbol, attr_name, column)
    unless self.class.read_methods.include?(attr_name) ||
           self.class.extended_attributes.keys.include?(attr_name.to_sym)
      define_read_method_without_enhances(symbol, attr_name, column)
    else
      old_method_name = "#{symbol}_without_extends"
      define_read_method_without_extends(old_method_name.to_sym, attr_name, column)
      evaluate_read_method attr_name, "def #{symbol}; enhance(#{old_method_name}, '#{attr_name}'); end"
    end
  end
  alias_method :define_read_method, :define_read_method_with_extends

  # Override read_attribute for first call (pre-edge)
  # or when AR::Base.generate_read_methods = false
  alias_method :read_attribute_without_extends, :read_attribute
  def read_attribute_with_extends(attr_name)
    enhance(read_attribute_without_extends(attr_name), attr_name)
  end
  alias_method :read_attribute, :read_attribute_with_extends

  private

  # Enhance the attribute as appropriate
  def enhance(value, attr_name)
    mod = self.class.extended_attributes[attr_name.to_sym]
    if mod && !value.nil?
      value.dup.extend(mod)
    else
      value
    end
  end
end

An alternate way to implement this pattern would have been to use composed_of and to create a FormattedText class that extend String. That may be better for performance reasons (as it seems that it would simplify method lookup), but strikes me as being slightly less Rubylike.

What do you think?

Teach Capistrano to Deploy From a Tag or Branch

2007-03-08T00:00:00-08:00

Teach Capistrano to Deploy From a Tag or Branch

There comes a time in the life of many an application where it becomes more or less stable. And once it becomes stable, it also becomes boring because the changes that a developer really wants to make are bound to make it unstable, and thus unsuitable for deployment.

An oft-used approach to this situation is to create a stable maintenance branch (i.e. application/branches/1.0) and create tags for point releases (i.e. application/tags/1.0.6), continuing on the trunk with potentially destabilizing work.

Out of the box, Capistrano only supports deploying from the tip of the trunk. The snippet below demonstrates how to deploy from a branch (to a QA or staging environment) or tag (releases to production should always have tags). Stick it in your config/deploy.rb.

set :base_repository, "http://svn.mojodna.net/repository/#{application}"
if variables[:tag]
  set :repository, "#{base_repository}/tags/#{variables[:tag]}"
elsif variables[:branch]
  set :repository, "#{base_repository}/branches/#{variables[:branch]}"
else
  set :repository, "#{base_repository}/trunk"
end

With this modification in place, you can now deploy from a branch:

$ RAILS_ENV=qa cap deploy -Sbranch=1.0

Or from a tag:

$ RAILS_ENV=production cap deploy -Stag=1.0.6

Go forth and revel in your instability!

Classloading in Rails

2007-02-12T00:00:00-08:00

Classloading in Rails

Occasionally, I see code similar to the following used as a shortcut to load all model objects. It’s usually in a Rake task or somewhere that follows a “global require” by looping through all available objects and doing something with them. It’s simple, straightforward, and a logical follow up to shorthand like ["post", "user"].each { |m| require m }. However, it often sits out there like a dormant volcano, with a wide variety of unpredictable behavior waiting to erupt.

In our case, we’d been using said snippet to help test out memcached; we would occasionally be handed serialized objects of unrecognizable types; the source of the problem turned out to be that the object’s class definition had not yet been loaded. We also discovered that finders in intermediate subclasses (we use STI extensively) would occasionally fail to find instances of subclasses, again because the subclasses had not been loaded (and the intermediate subclass was unable to provide a complete list of its subclasses).

# load all models explicitly
Dir.glob(File.join(RAILS_ROOT,"app","models","*.rb")).each do |rbfile|
  require rbfile
end

There are a couple of problems with this approach. It doesn’t guarantee that a model will only be loaded once–classes may have already been loaded through some external mechanism. Alternately, dependencies may be loaded while loading a specific class. In both cases, the above code may cause classes to be loaded multiple times (which can cause bizarre behavior). It also doesn’t handle namespaced models correctly.

# force loading (but not reloading) of all models by saying their name
# Adapted from PragDave's Annotate Models plugin.
model_path = File.join(RAILS_ROOT, "app", "models")
Dir.glob(File.join(model_path,"**","*.rb")).each do |m|
  class_name = m.sub(model_path + File::SEPARATOR, '').sub(/\\.rb$/, '').camelize
  klass = class_name.split('::').inject(Object){ |klass,part| klass.const_get(part) } rescue nil
end

This approach is considerably better; it supports namespaced models and defers the actual classloading to Rails (note the lack of an explicit require), which prevents classes from being loaded multiple times.

Anything to avoid time-eating debugging sessions, like the one described in Rails Sometimes Eats Class Load Errors. In our case, it had to do with ActiveMessaging and classes silently failing to load outside the development environment. The fix wound up being requiring config/messaging.rb before the block above. The lesson learned is to make sure that any dependent resources for models are loaded first and to look for them if things are acting strangely.

Update: The above primarily relevant to Rails 1.1. 1.2 exhibits similar behavior, but it gets even weirder, as the class loading mechanism was rewritten. There, you’ll find that STI works properly the first time (because all classes have been explicitly loaded), but if cache_classes is on (the default in development mode), they’ll be unloaded after the first request and thus will fail to work on subsequent requests.

Our rather hackish solution is to trigger a before_filter in ApplicationController when cache_classes is determined to be on (when Dependencies.mechanism == :load):

before_filter do
  model_path = File.join(RAILS_ROOT, "app", "models")
  Dir.glob(File.join(model_path,"**","*.rb")).each do |m|
    class_name = m.sub(model_path + File::SEPARATOR,'').sub(/\\.rb$/,'').camelize
    klass = class_name.split('::').inject(Object){ |klass,part| klass.const_get(part) } rescue nil
  end
end if Dependencies.mechanism == :load

Searchable: Annotation-Driven Indexing and Searching with Lucene

2006-10-02T00:00:00-07:00

Searchable: Annotation-Driven Indexing and Searching with Lucene

Overview

Searchable is a toolkit for Lucene that harnesses the power of annotations to specify what properties to index and how to treat them.

Basics

Searchable requires JDK 1.5 and Lucene 1.9+.

At its core, Searchable provides a set of annotations that can be used to instruct its indexer how to deal with properties present in an annotated bean. It can also be used without the annotations (by writing a custom Indexer/Searcher combination). However, Searchable really shines when custom Indexers and Searchers are combined with the annotations.

Searchable 0.8-SNAPSHOT can use Apache Solr for indexing and searching via SolrIndexer and SolrSearcher (SchemaGenerator will create an appropriate Solr schema for you; some tweaks may be necessary).

Searchable is available on GitHub here: http://github.com/mojodna/searchable Do with it what you will.

Please see the examples below for pointers on usage.

Annotations

In order for the annotations to have any effect, they must annotate a class that extends Searchable. With the exception of @DefaultFields, they must all be placed on the reader method of a property (getXXX()). Unlike traditional Annotation behavior, they may be placed on an interface method or a method that is overridden.

@ID

This allows a property other than id to be specified as the id field. There should only be one per class hierarchy.

e.g.:

@ID
public Integer getKey() { ... }

@Indexed

This specifies that a field should be indexed. The following attributes are available:

aliases - Array of aliases to also use as field names. Default: none.
boost - Boost factor (as a float) for this field. Default: 1.0.
name - Field name to use for this property. Default: property name.
nested - Whether to index this field in a nested context (i.e. a Searchable as a property of another Searchable). Default: false.
stored - Store this property in the index. Default: false.
storeTermVector - Store term vectors for this field. Default: false.
tokenized - Tokenize the value of this property before adding it to the index. Default: true.

e.g.:

@Indexed(boost=2.0F)
public String getName() { ... }

@Stored

This specifies that a field should be stored in the index. In typical usage, this should not be necessary, although you may find that it comes in hand from time to time. The following attributes are available:

aliases - Array of aliases to also use as field names. Default: none.
name - Field name to use for this property. Default: property name.
nested - Whether to store this field in a nested context (i.e. a Searchable as a property of another Searchable). Default: false.

e.g.:

@Stored
public String getDescription() { ... }

@Sortable

This specifies that a custom Keyword field should be created to use for sorting results in Lucene. The following attribute is available:

nested - Whether to create a sorted field for this property in a nested context (i.e. a Searchable as a property of another Searchable). Default: false.

e.g.:

@Indexed
@Sortable
public String getName() { ... }

@DefaultFields

This contains an array of field names that should be used as the default list. If this annotation is not present and no fields are specified when searching, the default is to use all fields present in the index.

Used during the search process.

e.g.:

@DefaultFields( { "name", "address.city" } )
public class Person implements Searchable { ... }

@Excerptable

This specifies that the annotated property should be used when creating a search extract. NOTE: The resultant object must be hydrated in your code before it can be excerpted. In many cases, this involves reloading the object from Hibernate Session by using session.load().

e.g.:

@Indexed
@Excerptable
public String getDescription() { ... }

Extension Points

AbstractSearcher, AbstractMultiSearcher, AbstractIndexer, and AbstractBeanIndexer are provided as abstract base classes with the majority of necessary functionality provided as protected methods. AbstractSearcher exposes multiple signatures for certain methods that allow alternate implementations of Lucene IndexReaders and Searchers. AbstractMultiSearcher makes use of these to implement cross-index searching (an example is provided below), but a similar approach could be used to implement remote searching.

setAnalyzer(Analyzer) and setIndexPath(String) should be used to override the default behavior of Searchable. Both are best called in your constructor.

getIndexReader(), getIndexModifier(), and getIndexSearcher() provide shared access to IndexReaders, IndexModifiers, and IndexSearchers over the index specified using setIndexPath(String). In certain circumstances, you may wish to override these methods to provide alternate implementations (a MultiSearcher for example; if you wish to provide a RemoteSearchable, you must define an additional method, as getIndexSearcher() returns a Searcher, not a Searchable).

Searcher and Indexer are provided as interfaces that may be extended to expose additional functionality to your application in a generic fashion.

Result and ResultImpl have been split into an interface and an implementation in order to hide the add(Result) and replace(Result, Result) methods as well as to provide additional flexibility for alternate implementations. One such alternate implementation would also implement DisplayTag’s PaginatedList.

AbstractResult is made available as a base class suitable for extension by objects that implement Result and are not required to extend anything else.

Batch Indexing

BatchIndexer extends the Indexer interface by introducing three methods: flush() (to be implemented by the indexer), setBatchMode(boolean), and isBatchMode(), of which the latter two are provided by the indexing infrastructure. flush() is executed during close() immediately before the index is optimized. A typical implementation calls flushDeletes(). which flushes any document deletions that had previously been queued (rather than flushing them immediately, as in a non-batch indexer). The hybrid example below demonstrates a BatchIndexer in action.

Limitations

AbstractSearcher does not yet support default field arrays as arguments to the various doSearch() methods.

The @DefaultFields and @Excerptable annotations are only available on objects that implement Searchable. Ideally, they would be available for any @Result, as there’s nothing that necessarily limits their use to Searchables.

Fields that are stored in the index are returned in a storedFields Map (specified in the Result interface) regardless of whether or not the names correspond to properties on the object being returned. In the future, Lucene could be used as a persistence tool of sorts by reconstituting the object as much as possible. Thus, if all properties of an object were indexed (and stored), the object could be fully reconstituted and no external persistence mechanism would be necessary. The trade-off is index size, so this would not be feasible for large datasets.

Examples

Simple Example

This example demonstrates how to use Searchable without using the annotations.

AddressIndexer.java:

/**
 * Indexes Addresses.
 */
public class AddressIndexer extends AbstractIndexer implements Indexer {
    public AddressIndexer() {
        // use /tmp/addresses as the index path
        setIndexPath("/tmp/addresses");
    }

    public void add(Address address) throws IndexingException {
        // create a document with "address" as the type
        Document doc = createDocument( "address", address.getId() );

        // add fields for each property of Address
        // implementation of Address is left to your imagination
        doc.add( Field.UnStored("street", address.getStreet() ) );
        doc.add( Field.UnStored("city", address.getCity() ) );
        doc.add( Field.UnStored("state", address.getState() ) );
        doc.add( Field.UnStored("zip", address.getZip() ) );

        // save the document to the index
        save( doc );
    }

    public void delete(Address address) throws IndexingException {
        delete( "address", address.getId() );
    }
}

AddressSearcher.java:

/**
 * Searches Addresses.
 */
public class AddressSearcher extends AbstractSearcher implements Searcher {
    public AddressSearcher() {
        // use /tmp/addresses as the index path
        setIndexPath("/tmp/addresses");
    }

    public ResultSet search(String query) throws SearchException {
        return doSearch( query );
    }
}

AddressIndexTest.java:

// ...
public void test() throws Exception {
    AddressIndexer indexer = new AddressIndexer();
    indexer.add( makeAddress() );
    indexer.close();

    AddressSearcher searcher = new AddressSearcher();
    ResultSet rs = searcher.search("city:Cambridge");
    Result result = rs.iterator().next();
    // Address does not implement Result, nor was it indexed with its
    // fully qualified class name, so the result is a GenericResult
    assertTrue( result instanceof GenericResult );
    GenericResult gr = (GenericResult) result;
    assertEquals( "address", gr.getType() );
}

Simple Annotation-Driven Example

This example demonstrates the basics of annotation-driven indexing.

SearchableBean.java:

/**
 * Implementation of Searchable to be indexed.
 *
 * Extends AbstractResult to avoid needing to implement Result methods
 * (inherited from Searchable).
 */
public class SearchableBean extends AbstractResult implements Searchable {
    private Integer id;
    private String name;

    public SearchableBean(Integer id, String name) {
        this.id = id;
        this.name = name;
    }

    public Integer getId() {
        return id;
    }

    public void setId(Integer id) {
        this.id = id;
    }

    @Indexed
    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}

IndexManager.java:

// ...
public void index(SearchableBean bean) throws IndexingException {
    BeanIndexer bi = new BeanIndexer();
    bi.add( bean );
    bi.close();
}

public ResultSet search(String query) throws SearchException {
    BeanSearcher bs = new BeanSearcher();
    // searching will attempt to reconstitute objects and set their ids
    // this allows the calling layer to know a) the type and b) the id
    // in order to refresh it properly
    return bs.search( query );
}

IndexManagerTest.java:

// ...
public void testSearch() throws Exception {
    IndexManager im = new IndexManager();
    im.index( new SearchableBean() );
    ResultSet rs = im.search("name:seth");
    assertEquals( 1, rs.size() );
    Result result = rs.iterator().next();
    assertTrue( result instanceof SearchableBean );
    assertEquals( 1, result.getId() );
    // name has not been set
    assertNull( result.getName() );
}

Hybrid Example

This example demonstrates how to use Searchable in a hybrid mode. TeapotIndexer can operate in batch mode.

Teapot.java:

/**
 * A representation of a teapot.  When searching the Teapot index, only the
 * "name" and "description" fields will be searched by default (if
 * @DefaultFields were not specified, all indexed fields would be searched).
 */
@DefaultFields( {"name", "description" } )
public interface Teapot extends Searchable {
    /**
     * Name of the Teapot.  This serves as the id, is indexed, and can be
     * sorted by.  This is the only field that is indexed when a Teapot is
     * nested in another object (a TeaSet, for example).
     */
    @ID
    @Indexed
    @Sortable
    public String getName();

    /**
     * Color of the Teapot.  Aliased to "colour" for Brits.
     */
    @Indexed(aliases="colour", nested=false)
    public String getColor();

    /**
     * Material the Teapot is made of.  Indexed and stored.
    @Index(stored=true, nested=false)
    public String getMaterial();

    /**
     * The type of tea this Teapot contains.  Not searchable, but stored.
     */
    @Stored(nested=false)
    public String getTeaType();

    /**
     * Description.  Indexed and used when creating an excerpt.
     */
    @Indexed(nested=false)
    @Excerptable
    public String getDescription();
}

TeapotIndexer.java:

/**
 * Indexes teapots.
 */
public class TeapotIndexer extends AbstractBeanIndexer implements BatchIndexer<Teapot> {
    public TeapotIndexer() {
        // use /tmp/teapots as the index path
        setIndexPath("/tmp/teapots");
    }

    public void add(Teapot tp) throws IndexingException {
        Document doc = doCreate( tp );

        processBean( tp );
        postProcessTeapot( tp );

        // save the document to the index
        save( doc );
    }

    public void delete(Teapot tp) throws IndexingException {
        doDelete( tp );
    }

    public void flush() throws IndexException {
        // flush any pending deletes
        flushDeletes();
    }

    protected Document postProcessTeapot(Teapot tp) {
        // add an "owner" field not specified by the teapot
        doc.add( Field.UnStored("owner", "Nathan") );
    }
}

TeapotSearcher.java:

/**
 * Searches teapots.
 */
public class TeapotSearcher extends AbstractSearcher implements Searcher<Teapot> {
    public TeapotSearcher() {
        // use /tmp/teapots as the index path
        setIndexPath("/tmp/teapots");
    }

    public ResultSet<Teapot> search(String query) throws SearchException {
        return excerpt( refresh( doSearch( query ) ) );
    }

    protected ResultSet<Teapot> load(ResultSet<Teapot> results) {
        ResultSetImpl rs = (ResultSetImpl) results;
        for (Teapot tp : results) {
            // session is a HibernateSession
            // your actual implementation may involve a DAO
            // replaces stub Teapot with a fully loaded one
            // ResultSetImpl handles transferring of Result properties
            rs.replace( tp, (Teapot) session.load( tp ) );
        }
        return results;
    }
}

TeapotTest.java:

// ...
public void test() throws Exception {
    BatchIndexer<Teapot> indexer = new TeapotIndexer();
    // run TeapotIndexer in batch mode
    indexer.setBatchMode( true );
    indexer.add( makeTeapot() );
    indexer.close();

    Searcher<Teapot> searcher = new TeapotSearcher();
    ResultSet<Teapot> rs = searcher.search("material:china OR wireframe");
    Iterator<Teapot> i = rs.iterator();
    Teapot firstResult = i.next();
    Teapot secondResult = i.next();
    assertEquals( "china", firstResult.getMaterial() );
    assertTrue( secondResult.getDescription().contains("wireframe") );
    assertTrue( secondResult.getSearchExtract().contains("wireframe") );
}

Searching Multiple Indexes Example

This example demonstrates how to use Searchable to search multiple indexes for different types of objects. This assumes a 1 class/index breakdown, but that’s not strictly necessary.

AddressAndTeapotSearcher.java:

/**
 * Searches Addresses and Teapots.
 */
public class AddressAndTeapotSearcher extends AbstractMultiSearcher implements Searcher {
    /**
     * Constructs this as a MultiSearcher for Addresses and Teapots.  The order and length
     * of both arrays must be equivalent.
     */
    public AddressAndTeapotSearcher() {
        super( new String[] { "/tmp/addresses", "/tmp/teapots"},
               new Class[] { Address.class, Teapot.class } );
    }

    /**
     * Searches the teapot index using default fields determined by @DefaultFields
     * annotation on the Teapot.  Also searches the address index using all available
     * fields as defaults.  If Address implemented Searchable and contained a
     * @DefaultField annotation, those fields would be used instead.
     */
    public ResultSet search(String query) throws SearchException {
        return doSearch( query );
    }
}

Sprout: Annotation-Powered Simplicity for Struts

2005-11-30T00:00:00-08:00

Sprout: Annotation-Powered Simplicity for Struts

Overview

Sprout aims to significantly simplify development with Struts by reducing the amount of configuration required through the use of annotations and sensible defaults.

Basics

Sprout requires JDK 1.5. Without that, you’re out of luck.

Sprout is an extension of a Struts MappingDispatchAction, which allows for multiple actions to be defined within the same Action class. In this case, the name of the method is mapped to the URL (after converting method names; methodName is exposed as /method_name.do). Paths are determined based on the package name of the action. For example, net.mojodna.sprout.action.HomeAction corresponds to /, while net.mojodna.sprout.action.example.ExampleAction corresponds to /example/.

Sprout also uses a custom RequestProcessor–SproutRequestProcessor, which extends Spring’s DelegatingRequestProcessor. This means that you can specify dependencies within your actions using setter-injection.

Sprout is also completely backward-compatible with legacy Struts applications.. It was built for use in a legacy Struts application; many of the older Actions are untouched–new development is done using Sprout (I often find myself removing action-mappings while adding new functionality).

Sprout is available on GitHub here: http://github.com/mojodna/sprout Do with it what you will.

Annotations

All annotations are optional.

@FormName

Allows the developer to override the name of the form-bean (defined in struts-config.xml) used for this method. This is equivalent to setting the name attribute within an action mapping.

Defaults to ${action-name}Form; e.g. for AdminAction the default ActionForm name would be AdminActionForm.

@Forward

Specifies additional forwards. Multiple forwards may be specified by providing arrays as arguments to name, path, and redirect. redirect defaults to false.

A default redirect is provided; the key is Sprout.FWD_SUCCESS and the path is the converted path + .jsp. E.g., AdminAction.methodName() corresponds to method_name.jsp.

e.g. @Forward(name=”failure”, path=”/failure.jsp” redirect=”true”)

@Input

This annotation is required if this action is validating the output from a different action. In that case, the argument to @Input should be the path to the JSP containing the form whose input is being validated.

e.g @Input(“login.jsp”) if this is not login() and the action that initiated this request is login().

@Scope

Specifies the scope attribute for the generated action mapping. This exists primarily for completeness; it is likely that you may never use this annotation.

As with struts-config.xml, the default is request.

@Validate

Specifies the validate attribute for the generated action mapping. Set this to true if you desire the output of this action to be validated. For this to have any effect, you must have specified rules in validator-rules.xml.

Sprout does not contain anything to ease the actual validation process at this time.

Example

src/java/net/mojodna/sprout/action/example/ExampleAction.java:

// URL should be /example/*
package net.mojodna.sprout.action.example;

// ...

public class ExampleAction extends Sprout {
  // overrides Sprout.index()
  public ActionForward index( ... ) {
    // do something

    // redirect to index.jsp
    return mapping.findForward( FWD_SUCCESS );
  }
}

src/java/applicationContext.xml:

...
<bean name="ExampleAction" class="net.mojodna.sprout.action.example.ExampleAction" singleton="true" />
...

src/web/WEB-INF/struts-config.xml:

...
<form-bean name="ExampleActionForm" type="org.apache.struts.validator.DynaValidatorForm">
  <form-property name="..." type="..." />
  ...
</form-bean>

<!-- No action-mappings!!! -->
<action-mappings />

<!-- Define an alternate RequestProcessor -->
<controller processorClass="net.mojodna.sprout.SproutRequestProcessor" />

<!-- Sprout plug-in -->
<plug-in className="net.mojodna.sprout.SproutAutoLoaderPlugIn" />
...

Shorthand

Index Actions

Sprout contains an index() method to speed up the process of getting something working. To use this, subclass Sprout (no methods necessary), register the action-bean in applicationContext.xml and create a corresponding index.jsp. When you need to add logic to the action, override index() in your Sprout sub-class and add it there.

DynaActionForms

Helper methods have been added to ease development using DynaActionForms.

String key = "foo";
String value = "bar";
// returns a String
f( key ) == ((DynaActionForm) form).getString( key );

// returns an Object
F( key ) == ((DynaActionForm) form).get( key );

// sets a value
s( key, value ) == ((DynaActionForm) form).set( key, value );

ActionMessage handling

Sprout contains adaptations to the traditional way Struts handles ActionMessages. getMessages() and getErrors() have been modified to store and retrieve messages from the session rather than the request. This means that messages and errors will be displayed (and subsequently cleared) on the next invocation of <html:messages /> or a variant (such as <ui:notifications />), regardless of whether either of the get methods have been called or if the invocation occurs during a separate request.

Sample message / error handling code (within an Action):

ActionMessages msgs = getMessages( request );
ActionMessages errors = getErrors( request );

// Add a message
msgs.add( ... );

// Add an error
errors.add( ... );

// Save messages and errors
saveMessages( request, msgs );
saveErrors( request, errors );

<ui:notifications /> (src/web/WEB-INF/tags/ui/notifications.tag) is an alternative tag file that can be modified for your use. The primary difference between <html:messages /> is that it will display both messages and errors (and will discriminate between them, allowing you to style them differently depending on your application’s needs).

Servlets and Taglibs

Spring’s Struts integration lacks the ability to autowire servlets and taglibs. Sprout contains classes (Sproutlet and SproutTag) that can be subclassed to provide auto-wiring capability. They will be wired during their initialization process.

These support classes only support the byName auto-wiring mechanism.

Q + A

Q: Why the Spring dependencies?
A: I’m already using Spring. There’s a good chance that you are as well. Removing the dependency makes reflection upon the registered actions (or finding them in the first place) significantly more difficult.

Introducing Sprout - Pure annotation goodness

2005-10-31T00:00:00-08:00

Introducing Sprout - Pure annotation goodness

Summary: Sprout. Check it out.

I have a Struts application that I work on every day. It has been a Struts application for about 3 years. It will likely continue to be a Struts application for the foreseeable future (it’s now 2009 and it’s a Rails app). I imagine some of you may be in the same position.

Ever since I started working on web applications in Java, I’ve been poking around the other web frameworks that are out there. WebWork is interesting (especially now that it’s to merge with Struts). Spring MVC is promising, if only because I like having my services wired together and to my actions. Tapestry seemed too complex, and SiteMesh, while not a framework, gave me everything that Tapestry was promising (granted, I haven’t looked at Tapestry a whole lot and should probably have another look). Wicket and especially JSF, ick. The web is not a client application and does not lend itself well to the Swing approach to components and widgets (yet). JSF, in particular, strikes me as great for both sides of the Toolbuilder / Application developer divide, but the overhead involved in creating custom components (a requirement for me, as I still like reinventing my HTML constructs) is simply too great. JSF in its optimal form also takes an event-based view of the web, which just doesn’t work so well yet (POSTs everywhere!). This isn’t about Java frameworks; this is about Struts.

Enter Ruby on Rails. Nifty. It fits the small development team model much better (I am a development team of 1; Struts and its ilk assume and thrive in an environment where tasks can and are divided up, such as the one my app was originally written in). It was not created as a framework, but more extracted from an application because it worked and made its creator’s life simpler. Convention over configuration (rather than conventions AND configuration). Embedded opinions that just happen to jive with my own view of the world. Looking at Rails and gaining an understanding of how it fits together changed the way I think about web application development. But this isn’t about Rails.

I have a Struts application that I work on every day. I have a minor Rails application that I rarely work on. I like Ruby. But I also use Java.

I recently spent some time focused on making Struts work for me rather than accepting the tedium of its status quo. Out grew a Sprout. The extract of something that makes my life easier. It might do the same for yours. Let me know.

Update: Richard Harms took this work a couple years after I abandoned it and maintained it for a year or so as a Google Code project. It lives on GitHub for posterity (and portfolio) and includes the work that Richard contributed.

Ongoing MBTA Work

2005-04-21T00:00:00-07:00

Ongoing MBTA Work

Update: I haven’t worked on this in many moons and it has ceased to work. Here is a currently working MBTA map.

The coverage area has increased to cover Boston, Charlestown, South Boston, and Brookline (the 4 downtown maps). Rather than creating the slices for tiles by hand in Photoshop, I whipped up a PHP script to do it for me - slicer.php (requires GD w/ GIF support, an increased memory_limit in php.ini, and a GIF version of full.psd (same dimensions)).

I added zoom level 1 by importing the PDFs as 3500px wide images and providing alternate coordinates to the slicer. Hint: it’s much faster to work with them after converting them to Indexed Color (via RGB, as they get imported as CMYK) (and obviates the need to Export to Web).

Continuing to map the MBTA

2005-04-20T00:00:00-07:00

Continuing to map the MBTA

Update: I haven’t worked on this in many moons and it has ceased to work. Here is a currently working MBTA map.

Ok, so it’s not that useful right now, since it only covers a small part of Cambridge at one zoom level. And since I don’t really have the time to keep working on it right now, I’ll post what I’ve got so anyone else can continue.

Maps in PDF
Tiles so far (x=5378, y=-740 (top left) to x=5386, y=-733 (bottom right)) (these are 128x128 each)
Tile server
Greasemonkey script (change mbta.baseURL to your own copy of the tiles)
Cambridge/Brookline/Charlestown/Boston, rasterized and stitched together (12MB)

The 4 downtown maps work best when rasterized to a 1710px wide canvas (before cropping) to create the tiles for zoom=2. I experimented with different sizes and even figured out the pixel/mile ratio (more below), but that didn’t help.

The PDFs are nice enough that it should be possible to create tiles for multiple zoom levels.

The MBTA also has a nice Trip Planner that could potentially be scraped to provide transfers on the sidebar (where driving directions would usually be placed). It appears to use lat/lon pairs internally, which is helpful.

MBTA Maps

2005-04-19T00:00:00-07:00

MBTA Maps

Update: I haven’t worked on this in many moons and it has ceased to work. Here is a currently working MBTA map.

Following on the heels of Adrian Holovaty’s work with Chicago Transit Authority maps, I dug around on the MBTA’s website (Boston’s T) for some decent maps. These are very clean and to-scale, so they were ripe for a Boston version.

After some fiddling around, I discovered that the Cambridge / Somerville inset appears to correspond with Google’s tiles when the PDF is imported into Photoshop with a width of 1710px. After some slicing and dicing, I give you:

This is a screenshot of the Greasemonkey version (with minor alterations to Adrian’s).

Building on top of some experimentation I’d done last week with Google Maps stand-alone, I made a non-Greasemonkey version by adding in a hook that effectively runs the Greasemonkey script. The default zoom-level is wrong, and my quick and dirty tile-serving code uses redirects (slowing things down and not handling missing tiles gracefully), so I’ll post that later once I get a chance to clean it up.

Update: I don’t have enough time to keep going on my own, so I’ve posted my work and a basic knowledge dump.

Update: Ok, so I found a bit more time and added zoom=1 and zoom=3 (the latter with the bigger map, thus covering the suburbs, though the North Shore is a bit off). Adding additional zoom levels (zoom=4, zoom=5?) is probably worthwhile to get more of an overview. Beyond that, what’s next is probably merging maps together to increase the coverage area and using the insets for additional zoom levels.

Digressions of a Drive-by Hacker

CSSpring Cleaning

CSSpring Cleaning

Things to Improve

Exploring OAuth-Protected APIs

Exploring OAuth-Protected APIs

Enter oauth-proxy

Access Tokens Sold Separately

A Concrete Example

Subscribing to Wordpress.com with Switchboard

Subscribing to Wordpress.com with Switchboard

Switchboard as a Framework

Switchboard as a Framework

A Simple Client

Jacks and Hooks

Standard Jacks

A More Complex Example, with Pandas

That’s it for now

switchboard : XMPP :: curl : HTTP

switchboard : XMPP :: curl : HTTP

Quick Start

“curl for XMPP?”

Switchboard background

What to do?

Configuration

Roster Manipulation

Probing and Discovery

PubSub

PEP (Personal Eventing Protocol)

More

Getting Help

Extending

How to Help

Updating Ruby Consumers and Providers to OAuth 1.0a

Updating Ruby Consumers and Providers to OAuth 1.0a

Updating Ruby OAuth Consumers to 1.0a

Updating Ruby OAuth Providers to 1.0a

Accepting oauth_callback During the Request Token Phase

Generating An oauth_verifier During the Authorization Phase

Verifying Access Token Exchanges

Summary

An Idiot's Guide to OAuth 1.0a

An Idiot’s Guide to OAuth 1.0a

What Happened?

So, What Changed?

Services Supporting OAuth 1.0a

My (public) Git Workflow

My (public) Git Workflow

Let’s Go

Determining “Active” and “Authoritative” Forks

Some Rules of Thumb

An Exception

Maintaining a Project

Aside: Complexity and / or Maturity

Now What?

My (work) Git Workflow

My (work) Git Workflow

A Not-So-Brief Introduction (with code!)

Bonus Points

The Sizzle

The Setup

Fixing a Bug / Implementing a Feature

Landing a Reviewed Patch

Working with Subversion Branches

Extra Credit

Extending ActiveRecord Attributes

Extending ActiveRecord Attributes

Teach Capistrano to Deploy From a Tag or Branch

Teach Capistrano to Deploy From a Tag or Branch

Classloading in Rails

Classloading in Rails

Searchable: Annotation-Driven Indexing and Searching with Lucene

Searchable: Annotation-Driven Indexing and Searching with Lucene

Overview

Basics

Annotations

@ID

@Indexed

@Stored

@Sortable

Enter `oauth-proxy`

Accepting `oauth_callback` During the Request Token Phase

Generating An `oauth_verifier` During the Authorization Phase