Although many of us wish we could consume the web through a magic programmer portal that shields us and our code from all the pointy angle brackets, the reality that is the legacy of HTML, Atom and RSS on the web leaves us little choice but to soldier on. So let’s take a look at what Ruby-colored armor is available to us as we continue our quest to slay the XML dragons.

Background

Historically, Ruby has had a number of options for dealing with structured markup, though oddly none have reached a solid consensus among Ruby developers as the “go to” library. The earliest available library seems to be Yoshida Masato’s XMLParser, which wraps Expat and was first released around the time that Expat itself was released, back in 1998. A pure Ruby parser by Jim Menard called NQXML appeared in 2001, though it never matured to the level of a robust XML parser.

In late 2001, Matz expressed his desire for out of the box XML support, but sadly, nothing appeared in Ruby’s standard library until 2003, when REXML was imported for the 1.8.0 release. After reading bike-shed discussions like this one on ruby-talk in November 2001, or this wayback-machine page from the old RubyGarden wiki, it’s not hard to see why. Meanwhile, other language runtimes, such as Python and Java, moved along and built solid, acceptable foundations, making Ruby’s omission seem more glaring.

But all was not lost: Ruby has always had a quality without a name that made it a great language for distilling an API. All that was needed was an infusion of interest and talent in Ruby, and a few more experiments and iterations.

Fast forward to the present time, and all those chips have fallen. We’ve seen evolution from REXML to libxml-ruby to Hpricot, and finally to Nokogiri. So, is the XML landscape on Ruby so dire? Certainly not, as you’ll see by the end of this article! While the standard library support for XML hasn’t progressed beyond REXML yet, state-of-the-art solutions are a few keystrokes away.

XML APIs

A big part of what makes XML such a pain to work with is the APIs. We Rubyists tend to have an especially low tolerance for friction in API design, and we really feel it when we work with XML. If XML is just a tree structure, why isn’t navigating it as simple and elegant as traversing a Ruby Enumerable?

The canonical example of API craptasticism is undoubtedly the W3C DOM API. For proof, observe the meteoric rise of jQuery in the JavaScript world. While it would be easy to fill an entire article with criticisms regarding the DOM, it’s been done before. (Incidentally, read the whole series of interviews with Elliotte Rusty Harold for a series of insights on API design, schema evolution, and more.)

Instead, we’ll take a brief exploratory tour of some Ruby XML APIs using code examples. Though some of the examples may seem trivially short, don’t underestimate their power. Conciseness and readability are Ruby’s gifts to the library authors and they’re being put to good use.

The libraries we’ll use for comparison are REXML, Nokogiri, and JAXP, Java’s XML parsing APIs (via JRuby).

Parsing

The simplest possible thing to do in XML is to hand the library some XML and get back a document.

REXML

require 'rexml/document'
document = REXML::Document.new(xml)

Nokogiri

require 'nokogiri'
document = Nokogiri::XML(xml)

Both REXML and Nokogiri more or less get this right. What’s also nice is that they both transparently accept either an IO-like object or a string. Contrast this to Java:

JAXP/JRuby

factory = javax.xml.parsers.DocumentBuilderFactory.newInstance
factory.namespace_aware = true # unfortunately, important!
parser = factory.newDocumentBuilder
# String
document = parser.parse(xml)
# IO
document = parser.parse(xml.to_inputstream)

In that familiar Java style, the JAXP approach forces you to choose from many options and write more code for the happy path. JRuby helps you a little bit by converting a Ruby string into a Java string, but needs a little help with intent for converting an IO to a Java InputStream.

XPath

Now that we’ve got a document object, let’s query it via XPath, assuming the underlying format is an Atom feed. Here is the code to grab the entries’ titles and store them as an array of strings:

REXML

elements = REXML::XPath.match(document.root, "//atom:entry/atom:title/text()",
                              "atom" => "http://www.w3.org/2005/Atom")
titles = elements.map {|el| el.value }

Nokogiri

elements = document.xpath("//atom:entry/atom:title/text()",
                          "atom" => "http://www.w3.org/2005/Atom")
titles = elements.map {|e| e.to_s}

Again, both REXML and Nokogiri clock in at similar code sizes, but subtle differences begin to emerge. Nokogiri’s use of #xpath as an instance method on the document object feels more natural as a way of drilling down for further detail. Also, note that both APIs return DOM objects for the text, so we need to take one more step to convert them to pure Ruby strings. Here, Nokogiri’s use of the standard String#to_s method is more intuitive; REXML::Text’s version returns the raw text without the entities replaced.

Unfortunately, doing XPath in Java gets a bit more complicated. First we need to construct an XPath object. At least JRuby helps us a bit here–we can create an instance of the NamespaceContext interface completely in Ruby, and omit the methods we don’t care about.

JAXP/JRuby

xpath = javax.xml.xpath.XPathFactory.newInstance.newXPath
ns_context = Object.new
def ns_context.getNamespaceURI(prefix)
  {"atom" => "http://www.w3.org/2005/Atom"}[prefix]
end
xpath.namespace_context = ns_context

Next, we evaluate the expression and construct the array titles:

JAXP/JRuby

nodes = xpath.evaluate("//atom:entry/atom:title/text()",
                       document, javax.xml.xpath.XPathConstants::NODESET)
titles = []
0.upto(nodes.length-1) do |i|
  titles << nodes.item(i).node_value
end

That last bit where we need to externally iterate the DOM API is particularly un-Ruby-like. With JRuby we can mix in some methods to the NodeList class:

JAXP/JRuby

module org::w3c::dom::NodeList
  include Enumerable
  def each
    0.upto(length - 1) do |i|
      yield item(i)
    end
  end
end

And replace the external iteration with a more natural internal one:

JAXP/JRuby

titles = nodes.map {|e| e.node_value}

This kind of technique tends to become a fairly common occurrence when coding Ruby to Java libraries in JRuby. Fortunately Ruby makes it simple to hide away the ugliness in the Java APIs!

Walking the DOM

Say we’d like to explore the DOM. Both REXML and Nokogiri provide multiple ways of doing this, with parent/child/sibling navigation methods. They also each sport a recursive descent method, which is quite convenient.

REXML

titles = []
document.root.each_recursive do |elem|
  titles << elem.text.to_s if elem.name == "title"
end

Nokogiri

titles = []
document.root.traverse do |elem|
  titles << elem.content if elem.name == "title"
end

Needless to say, Java’s DOM API has no such convenience method, so we have to write one. But again, JRuby makes it easy to Rubify the code. Note that our #traverse method makes use of our Enumerable-ization of NodeList above as well.

JAXP/JRuby

module org::w3c::dom::Node
  def traverse(&blk)
    blk.call(self)
    child_nodes.each do |e|
      e.traverse(&blk)
    end
  end
end

titles = []
document.traverse do |elem|
  titles << elem.text_content if elem.node_name == "title"
end

Pull parsing

All three libraries have a pull parser (also called a stream parser or reader) as well. Pull parsers are efficient because they behave like a cursor scrolling through the document, but usually result in more verbose code because of the need to implement a small state machine on top of lower-level XML events. They are best employed on very large documents where it’s impractical to store the entire DOM tree in memory at once.

REXML

parser = REXML::Parsers::PullParser.new(xml_stream)
titles = []
text = ''
grab_text = false
parser.each do |event|
  case event.event_type
  when :start_element
    grab_text = true if event[0] == "title"
  when :text
    text << event[1] if grab_text
  when :end_element
    if event[0] == "title"
      titles << text
      text = ''
      grab_text = false
    end
  end
end

Nokogiri

reader = Nokogiri::XML::Reader(xml_stream)
titles = []
text = ''
grab_text = false
reader.each do |elem|
  if elem.name == "title"
    if elem.node_type == 1  # start element?
      grab_text = true
    else # elem.node_type == 15  # end element?
      titles << text
      text = ''
      grab_text = false
    end
  elsif grab_text && elem.node_type == 3 # text?
    text << elem.value
  end
end

(Aside to the Nokogiri team: where are the reader node type constants?)

JAXP/JRuby

include javax.xml.stream.XMLStreamConstants
factory = javax.xml.stream.XMLInputFactory.newInstance
reader = factory.createXMLStreamReader(xml_stream.to_inputstream)
titles = []
text = ''
grab_text = false
while reader.has_next
  case reader.next
  when START_ELEMENT
    grab_text = true if reader.local_name == "title"
  when CHARACTERS
    text << reader.text if grab_text
  when END_ELEMENT
    if reader.local_name == "title"
      titles << text
      text = ''
      grab_text = false
    end
  end
end

Not surprisingly, all three pull parser examples end up looking very similar. The subtleties of the pull parser APIs end up getting blurred in the loops and conditionals. Only write this code when you have to.

Performance

At the end of the day, it comes down to performance, doesn’t it? Although the topic of Ruby XML parser performance has been discussed before, I thought it would be instructive to do another round of comparisons with JRuby and Ruby 1.9 thrown into the mix.

System Configuration

• Mac OS X 10.5 on a MacBook Pro 2.53 GHz Core 2 Duo
• Ruby 1.8.6p287
• Ruby 1.9.1p243
• JRuby 1.5.0.dev (rev c7b3348) on Apple JDK 5 (32-bit)
• Nokogiri 1.4.0
• libxml2 2.7.3

Here are results comparing Nokogiri and Hpricot on the three implementations  along with the JAXP version which only runs on JRuby (smaller is better).

The REXML results were over an order of magnitude slower, so it’s easier to view them on a separate graph. Note the number of iterations here is 100 vs. 1000 for the results above.

While these results don’t paint a complete picture of XML parser performance, they should give you enough of a guideline to make a decision on which parser to use once you take portability and readability into account. In summary:

• Use REXML when your parsing needs are minimal and want the widest portability (across all implementations) with the smallest install footprint.
• Use JRuby with the JAXP APIs for portability across any operating system that supports the Java platform (including Google AppEngine).
• Use Nokogiri for everything else. It’s the fastest implementation, and produces the most programmer-friendly code of all Ruby XML parsers to date.

(As a footnote, we on the Nokogiri and JRuby teams are looking for community help to further develop the pure-Java backend for Nokogiri so that AppEngine and other JVM deployment scenarios that don’t allow loading native code can benefit from Nokogiri’s awesomeness. Please leave a comment or contact the JRuby team on the mailing list if you’re interested.)

The source code for this article is available if you’d like to examine the code or run the benchmarks yourself. Keep an eye on the Engine Yard blog for an upcoming post on Nokogiri, and as always, leave questions and thoughts in the comments!

The difficult part of discussing Key-Value Stores stores today is that it’s a product area seeing rapid development and constant evolution. There are more interesting stores and libraries available than can easily be covered, even in a series like this. I could probably be writing posts every two weeks into next year without running out of subjects. So, alas, many things must be left undiscussed or underdiscussed. But let’s move on to the topics we can cover…

CouchDB

The first great Key-Value Store that isn’t going to get its own article is CouchDB. Apache’s CouchDB is a document-oriented database, like MongoDB. It, however, exposes a RESTful JSON based API that you address with a built in HTTP interface. Like MongoDB, it offers a schema free data store. CouchDB offers solid, built-in replication, and uses JavaScript as its query language. It is a powerful tool.

There are several Ruby libraries which can be used to facilitate using CouchDB. In the examples below, I have used CouchRest, which is based on CouchDB’s own couch.js library:

require 'rubygems'
require 'couchrest'
require 'yaml'

DBH = CouchRest.database!('exercise-log')

response = DBH.save_doc({
  :date => Time.now,
  :activity => ARGV[0],
  :duration => ARGV[1]})

stored_record = DBH.get(response['id'])
puts "Stored:\n#{stored_record.to_yaml}"

wyhaines$ ruby /tmp/couch1.rb
Stored:
--- !map:CouchRest::Document
duration: "97:34"
_rev: 1-eb6f6e3a3e2eae0cd99f3fcbc63d29d6
_id: 0d9e71f44b3e0d3a2013c282bbccb5a0
activity: pedaling
date: 2009/11/12 21:07:45 +0000

Like MongdoDB, one can store any set of keys/values together as a document in CouchDB, and then retrieve it later. CouchRest returns a response from the server that contains an id field, which can be used to retrieve the record that was just stored.

For more complex queries of the document store, one can use views. Views have a lot of power, because they are ultimately defined using JavaScript, but they don’t lend themselves to easy ad-hoc manipulation of the database.

DBH.save_doc({
  "_id" => "_design/query",
  :views => {
    :allkeys => {
      :map => "function(doc) { for (var word in doc) { if (!word.match(/^_/)) emit(word,doc[word])}}"
    }
  }
})

That inserts a view into the database that will be identified by query/allkeys. What a view does is defined by the JavaScript code  it contains. Once a view is inserted into CouchDB, using it is simple:

puts DBH.view('query/allkeys').to_yaml

That particular function was lifted shamelessly from the CouchRest README, and just has a couple terms renamed to make it a little more clear. The output:

---
total_rows: 3
rows:
- id: 0d9e71f44b3e0d3a2013c282bbccb5a0
  value: pedaling
  key: activity
- id: 0d9e71f44b3e0d3a2013c282bbccb5a0
  value: 2009/11/12 21:07:45 +0000
  key: date
- id: 0d9e71f44b3e0d3a2013c282bbccb5a0
  value: "97:34"
  key: duration
offset: 0

This is really just the tip of the iceberg with CouchDB/CouchRest; there’s a wealth of functionality. CouchDB views are implemented with map/reduce capability, which means you can use them to crunch some pretty complex problems on your data. Additionally, CouchRest provides a CouchRest::ExtendedDocument, which your own classes can inherit from. This lets you  easily create a Ruby model for your data, which is then transparently stored inside CouchDB.

class Exercise  "running", :date => Time.now, :duration => "23:44")

Dig into the CouchDB and CouchRest documentation if this looks interesting to you.

S3

I just wanted to briefly mention Amazon’s Simple Storage Service. It is, fundamentally, a simple HTTP accessible Key-Value Store that Amazon has turned into a service. Requests to S3 will have higher latency than requests to a locally hosted data store (and its response latency can be high too), but if you want a simple, robust store that will scale to as much data as you have to push at it, you might seriously consider S3.

Moneta

Moneta is a unified interface to a variety of different key-value type data stores. That is, the same code can be run against a variety of different backing stores, and it will just work. Moneta supports the following stores as of this posting:

• Basic File Store
• BerkeleyDB
• CouchDB
• DataMapper
• File store for xattr
• In-memory store
• Memcache store
• Redis
• S3
• SDBM
• Tokyo
• Xattrs in a file system

Consider this example, which, again, uses CouchDB:

irb(main):003:0> require 'moneta/couch'
require 'rubygems'
require 'yaml'
require 'moneta'
require 'moneta/couch'

cache = Moneta::Couch.new(:db => 'football')

cache['1a_final'] = {
  :where => 'Laramie; War Memorial Stadium',
  :when => "11:30 MST",
  :who => "Southeast Cyclones & Lingle-Ft. Laramie Doggers",
  :prediction => "SE Cyclones by 14"}

puts cache['1a_final'].inspect

wyhaines$ ruby /tmp/moneta1.rb
---
- prediction: SE Cyclones by 14
  when: 11:30 MST
  who: Southeast Cyclones & Lingle-Ft. Laramie Doggers
  where: Laramie; War Memorial Stadium

It works, very simply. If I want to change the code to use something else, like a file based store, it’s as simple as changing one line:

--- couch.rb    2009-11-19 15:00:07.000000000 -0700
+++ file.rb     2009-11-19 15:01:12.000000000 -0700
@@ -1,9 +1,9 @@
 require 'rubygems'
 require 'yaml'
 require 'moneta'
-require 'moneta/couch'
+require 'moneta/file'

-cache = Moneta::Couch.new(:db => 'football')
+cache = Moneta::File.new(:path => '/tmp/football')

 cache['1a_final'] = {
   :where => 'Laramie; War Memorial Stadium',

The rest of the code works without alteration. The Moneta API is designed to be very similar to that of Hash. It has a limited feature set, but the features it provides work identically across all of the supported platforms. For example, it doesn’t currently support iteration or partial matches. If your Key-Value Store needs are simple and you want something that can work with whatever store your users want to use, definitely check out Moneta; it’s a well written tool.

With that, we’ve reached the end of this series. It’s been fun to explore the unique features, as well as the threads that unify each of these different approaches to the problem, on a non-SQL key-value type data store. I hope that I’ve exposed you to new and useful tools.

The landscape of Key-Value Stores is changing rapidly, so it is difficult to stay fully informed all the time. For instance, just a couple days ago there was a blog post implementing a SQL front end for CouchDB. It’s done in Perl, but all it would take is an interested person and a little time, and you could have it in Ruby, too.

If you use a Key-Value Store system, or plan to, keep your eyes open for new developments, because you can bet that someone else will have something interesting next week or next month that may change the landscape again. As always, leave feedback in the comments, and thanks for reading!

Earlier this month, the Engine Yard crew took a quick trip around the states, stopping in five different cities to talk about Rails Performance in the Cloud. We visited Boston, Chicago, Austin, Los Angeles and Seattle, and met with a great crowd of technologists in each. We were joined by some of our favorite technology partners, who talked about cloud computing and their different performance-related products.

The Roadshow was a fantastic success; attendee feedback was overwhelmingly positive, and on our end, it was great to get out and meet folks in their home cities. Our partners enjoyed meeting existing customers, and for those who weren’t customers, we were able to present a uniquely well-rounded view of the ecosystem.

On to the point: the Roadshow was so fantastic, that we decided we ought do it one more time—this time, here at home, in San Francisco. I’m happy to announce the sixth iteration of the Rails Performance in the Cloud Roadshow, set for Wednesday, December 2nd, bright and early in the day.

We’ll be hosting right here at Engine Yard Headquarters in the SOMA part of San Francisco, and will be joined by our partners at New Relic, Pivotal Labs, Soasta and CVSDude. We’ll go from 8:15 in the morning until noon, and of course, breakfast will be served.

Space is limited, so reserve your seat today!

UPDATE: Challenge calculations submissions should be in the form of a RETWEET to @engineyard of the first message in your wase program from your home twitter account.  The final WASEpoint in your program listing should be @eycontest (so we can measure who finished first). You can register your wasepoints here.]

People often talk about the momentum of intangible things, but it’s always been pretty much impossible to define. No Longer! In this contest, you’ll be measuring the momentum of Rails. How will you accomplish this, you ask?

Well, since Rails is its community, if we can measure the mass, speed and direction of the Rails community, then (clearly!) we can establish its momentum. For the purposes of this contest, and to make things easy, we will use as a rough proxy that the Rails community is everyone following @dhh on twitter.

You must perform the following tasks to establish Rails momentum:

1) Establish the mass of the Rails community: defined as looking up the locations of all @dhh followers from Twitter, geo-coding their locations, and then multiplying the number of people in each location by the average body mass of an adult in that country.

2) Establish the current location of the Rails community: defined by taking the locations and body weights from step 1, and calculating the community’s center of mass (its centroid).

3) Establish the speed and direction of the Rails community: defined by taking its current location (calculated in Step 2) and comparing it to the origin of Rails. Rails 1.0 was released in Chicago on December 14th, 2005, so in the approximately 1,430 days since then, its location has moved. This means we can calculate its average speed and direction over the last four years.

Although we will expect the answer in the form of JSON object properties, the answer (in free text) to the contest would look like:

• Rails Momentum is 15 metric tons per hour with a bearing of 120 degrees. Its current location is latitude 38.898748 ° and longitude -77.037684°

Guidelines and Suggestions for Implementors

Please refer back to our earlier post for rules on how your entry must be structured and submitted. This description goes into details on the format of the input data and provides suggestions for WASEpoints and tips on what to avoid. A significant meta-challenge is agreeing on intermediate object formats with your fellow contestants! We have not specified what they should be.

1) Calculating the mass of the Rails community

The input data set has an array of all the twitter ID’s currently following @dhh. Twitter limits your API access to 150 calls per hour, unless you are a Twitter white-listed developer—so there is a clear opportunity for a white-listed developer to create a popular WASEpoint. If you’re not a white-listed developer, be prepared to collaborate with others to pool your twitter requests via WASEpoint chaining. We will make an exception to the standard WASEpoint rules here on “no state at a WASEpoint” and allow people to create WASEpoints for others that provide a cached location lookup for the location property of each follower.

Only about 50% of twitter user profiles have standard formatted locations in the form of [City, State], [State, City] [State] or [City, Country], [Country, City] [Country]. Location data that lacks this format should be discarded, and the corresponding followers ignored. There’s an opportunity here to write a WASEpoint that cleans and standardizes this data-set for others to use. We do not expect you to geocode imprecise location tags such as “The Midwest” or “Somewhere in the Clouds”.

There are many REST-accessible geocoding services on the web. Both Yahoo and Google have REST API’s and there are other responsive geocoding services with Ruby clients. We include a list of standardized countries and the average adult bodyweight as part of the input dataset for use in calculations. (It was actually quite surprising to see how many sources we had to go to to get this data, and it’s mostly only accurate for OECD countries—most countries simply contain weights based on an average BMI.) Bodyweights are in kilograms. This is another module of work that would make a nice WASEpoint.

2) Calculating the current location

Calculating a center of mass for a sphere like the earth would basically put Rails in the center of the earth, but we didn’t want to imply that Rails has gone to hell. Instead you are allowed to assume a flat-earth when calculating the centroid of the community, and the center of this flat mapping should be at Chicago’s longitude and the equator’s latitude. This latitude/longitude is provided as part of the input data set.

This puts the “edge” of this map approximately in western China, so India is to the “east” of Chicago, but Bangladesh would be “west” of Chicago. In this map, the center of mass for a developer in India and a developer in Bangladesh would not be on their border, it would be in the Caribbean. There is a clear opportunity here to write two WASEpoints: one that geocodes correctly and one that calculates centroids correctly.

(If anyone wants to tackle calculating a true 3D centroid with a projection back to the nearest point on the earth’s surface that would certainly be a strong candidate for third prize (best WASEpoint))

3) Establishing the speed and direction

Rails 1.0 was announced in Chicago (lat 41 54 long 87 39 ) on Dec 14th, 2005. If you establish that the center of the community is now in (for example) Albany, New York (lat 42 45 long 73 48), then it has traveled approximately 711 kilometers in 1,430 days, giving it an average land velocity of about 0.25 km/hr (unladen), and a final bearing of about 80°. For this part of the calculation we DO expect you to consider the earth spherical, but there are a number of handy math guides on the internets to help you calculate bearings from latitude and longitude data. This conversion alone could make a nice WASEpoint for others to use.

The Input Data Object

We have posted the input data for the challenge on web. We’re giving you two JSON files in UTF-8 format.

• TwitterList.json= An array of twitter id’s of everyone following @dhh

• WeightbyCountry.json = An array of Countries with their Bodyweight Values.

The Output Data Object

• We will accept output data in whatever data format your final WASEpoint emits, but it must be obvious to us what the momentum number is and what the bearing is. We would suggest the following json format however (with sample data):

{”RailsMomentum”:{”Momentum”:{”kgs per meter per second”:15},”location”:[{"latitude":333333,"longitude":333333}],”bearing”:{”degrees”:36,”minutes”:45,”seconds”:45}}}

Simple enough, right? Well, no one every said it was going to be easy! In fact, we tried to structure this, so you’d have to compete as part of a team! Good luck—we can’t wait to see how it goes!

Let’s start off with a simple, two-part application. Part 1 is a Sinatra app that puts JSON-serialized messages into an AMQP queue. Part 2 is a daemon that consumes those messages.

Here’s what a Gemfile for this application might look like:

gem 'json'
gem 'sinatra'
source 'http://gems.github.com'
gem 'famoseagle-carrot', :require_as => 'carrot'

gem 'eventmachine'
gem 'amqp'

The Sinatra app starts off with these lines:

# This makes sure the bundled gems are in our $LOAD_PATH
require File.expand_path(File.join(File.dirname(__FILE__), 'vendor', 'gems', 'environment'))

# This actually requires the bundled gems
Bundler.require_env

class MyApp < Sinatra::Base
  # stuff

The daemon starts like this:

# This makes sure the bundled gems are in our $LOAD_PATH
require File.expand_path(File.join(File.dirname(__FILE__), 'vendor', 'gems', 'environment'))

# This actually requires the bundled gems
Bundler.require_env

AMQP.start do
  # stuff

However, this loads too many gems. The Sinatra app synchronously publishes its messages using carrot, so it doesn’t need the EventMachine gem or the AMQP gem. Likewise, the daemon doesn’t serve HTTP requests, so it doesn’t need Sinatra, and it’s processing messages asynchronously using EventMachine, so it doesn’t need Carrot.

This isn’t the end of the world for this set of gems; it just makes all the Ruby processes use a little more memory than they otherwise might. However, if the Sinatra app uses one gem that defines Array#foo one way, and the daemon uses another gem that defines Array#foo an entirely different way, then each component must require only its own dependencies.

Fortunately, the bundler makes this really easy: just change the Gemfile to look like this:

gem 'json'

only :app do
  gem 'sinatra'
  source 'http://gems.github.com'
  gem 'famoseagle-carrot', :require_as => 'carrot'
end

only :daemon do
  gem 'eventmachine'
  gem 'amqp'
end

In the Sinatra app, change Bundler.require_env to Bundler.require_env(:app). This loads JSON, Sinatra, and Carrot, but not EventMachine or AMQP.

Similarly, in the daemon, Bundler.require_env becomes Bundler.require_env(:daemon).

But wait, it gets better!

Bundler.require_env can be called more than once with a different environment name each time. Consider tests (which, of course, you wrote first, right?): you want to run them. Now your Gemfile might look like this:

gem 'json'

only :app do
  gem 'sinatra'
  source 'http://gems.github.com'
  gem 'famoseagle-carrot', :require_as => 'carrot'
end

only :daemon do
  gem 'eventmachine'
  gem 'amqp'
end

only :test do
  gem 'rspec'
  gem 'webrat'
end

Then, in your spec, you start like this:

require 'my_app'       # load the Sinatra app so we can test it
Bundler.require_env(:test)    # get rspec and webrat in here

describe "the main page" do
  # stuff

Now each part of your app has its dependencies neatly contained. Each part only gets what it needs, and the different dependencies can’t step on each others’ toes.

Sweet.

Next week is going to be quite the week for Ruby in San Francisco, with QCon, RubyConf and JRubyConf. Engine Yard is excited to be participating in all three events, but there’s even more to tell you about!

With so many Rubyists in town, it only made sense to find a way to have you all over for drinks and socializing. As such, we’ve signed on to participate in the RubyConf week Startup Crawl. We’re working with our friends over at Scribd, along with a host of other local startups (Zendesk, Yammer, Justin.tv and more!) and opening our doors to everyone.

The Crawl will take place Friday evening, November 20th; there will be buses leaving from the RubyConf hotel starting at 5:30 and making trips back and forth until 11 p.m. We’ll have food, drinks, giveaways, and of course Engine Yard Cloud demos for all.

If you’re planning on taking the bus, be sure to sign up at the Startup Crawl site, so we can be sure there’s enough transportation for everyone. Otherwise, feel free to walk in any time after 5:30—we’d love to meet you!

Engine Yard customers have already been contacted via email about this issue. For Engine Yard Cloud customers, this patch will be automatically applied the next time you perform a deploy. All other customers should open a support ticket so that you can arrange an appropriate maintenance window with support.

Attendee James McElhiney beat me to the punch with his review, but I still wanted to take a few minutes to talk about how things went.

We started with Boston, and then moved to Chicago, Austin, Los Angeles and Seattle. Our partners Amazon Web Services, CVSDude, New Relic and Soasta joined us for the trip, meeting with various customers, developers and managers. The crowd was varied, highly technical, and despite the early-morning start, had great energy.

The Roadshow was all about performance, and between the presentations and audience questions, we covered close to everything you’d need to know to meet and exceed your performance goals. Some specifics:

Tom Mornini and I presented Engine Yard’s value proposition, along with a live demo of the Engine Yard Cloud platform. Tom focused on end-user performance and the very significant impact it can have on business metrics, while I went into detail on some performance metrics that should be relevant to all rails developers, highlighting how Engine Yard Cloud can help users scale in the cloud.

Next New Relic walked us through RPM, and how they use it internally to monitor, well, RPM. Self-referential coolness aside, it was a great way to see how a tool can help you identify the bottlenecks in your application, while still remaining easy to setup and maintain. When the first audience question started with “I’m signing up for your service even as I ask this…” we knew things were on the right track.

CVSDude’s Willie Wang presented on enterprise-grade source control management, and how their solutions can help achieve agility in deployments. This is a key component of performance optimization—agile deployments enable us to fix performance problems faster and deliver features to customers more effectively. With Git support in their near-term roadmap, CVSDude was a great addition for our Rails audience.

Many of the attendees were customers of Amazon AWS, and Mike Culver’s presentations described the value proposition of AWS very well. The Amazon team had a busy week—with newly announced services, instance types and a price reduction (which we at Engine Yard immediately passed along to our customers). As expected, a lot of the questions were around their new announcements, and Mike explained how each of these increased the value proposition of cloud computing even more.

Finally, Soasta’s Dave Murphy had a great presentation on load testing using Soasta Cloud Test. Some of his numbers were particularly impressive—seeing them use cloud infrastructure to deploy multi-million user load tests on live production environments really brought home the value of using the cloud model. Tools like CloudTest, in conjunction with Engine Yard, can make it trivial for users to test deployments that would have been impossible to test before.

For me personally, the roadshow was a great way to meet customers, developers and spend cycles understanding their needs better. The biggest takeaway for me was that customers really care about scalability, but realize that they add most value by focusing on user happiness. This was exactly the philosophy we used while designing Engine Yard Cloud—we want you to continue to focus on features that add value to your products, and leave all your scalability, performance and agility needs to the Engine Yard Cloud platform.

If you missed the Roadshow, be sure to register for our upcoming Engine Yard Cloud Demo Webinar; I’ll be walking through some of the topics covered at the Roadshow and taking questions.

We look forward to visiting more cities soon. Until then, enjoy the presentations, and send us your comments and feedback!

(Update: We think the rule-set below is now complete, but we still welcome any suggestions or tweaks that you might have.)

There will be three prizes.

• The first prize (a Motorola DROID and $1,000 of Engine Yard Cloud credit) goes to the person who completes our challenge task correctly first.
• Second prize (a DROID and $500 of cloud credit) goes to the person who builds the most popular WASE endpoint (the one used the most often in the most submissions).
• Third prize (a DROID) goes to the “best” WASE endpoint written in Ruby (as determined by us). The contest DROIDS are full price, non-contract-linked, US models.

How WASE Works

Why is WASE the worlds worst app server technology? Well, instead of a sane message bus like AMQP, WASE uses Twitter as its message bus. Instead of a proper message router, WASE uses a list of twitter accounts as its program listing. And instead of encapsulating data with each message, WASE messages only contain a reference to JSON objects or arrays at input and output location(s) specified by a bit.ly.

“This Sounds Like the World’s Worst App Server. Tell Me More.”

Well here is an example of how a sample computation might work. Let’s say @engineyard wants to take a JSON file containing an array of names, and get a list of  the top quartile of names after sorting the array. We know that there are two Twitter accounts (which we will henceforth call WASEpoints) @ey-sort and @ey-firsthalf that can be useful to us. We know @ey-sort takes an input data set, sorts it and outputs the result. We also know that @ey-firsthalf takes an input data set, and outputs the first “half” of the dataset. To perform a computation, we set up URI’s for the program listing, the input data and the output data, and kick off the computation with an appropriate message. (For those of you with dataflow or actor-based programming experience, WASE should look like a vague, but disreputable cousin.)

So let’s go through the message flow:

We’ll put our program listing at:

www.engineyard.com/top-quartile-sorted-list.json,  (bit.ly/7yQK6) whose body contents are a JSON array:

["@ey-sort", "@ey-firsthalf", "@ey-firsthalf", "@engineyard"]

We put our input data here:  www.engineyard.com/unsortedmegalist.json (bit.ly/3kl0xs)

And set up a location for our output data here:  www.engineyard.com/top25percentofmymegalist.json (bit.ly/2uhGcl)

Or to summarize the bit.ly’s,

Program listing: bit.ly/7yQK6 (read with a http: GET)

Output location: bit.ly/2uhGcl (written with a http: PUT)

Input location: bit.ly/3kl0xs (read with a http: GET)

To perform the computation, we’d simply send the following twitter message from our @engineyard account: @ey-sort #wase, 0, bit.ly/7yQK6, 1256850843, bit.ly/2uhGcl, bit.ly/3kl0xs

So the message format of a WASTE message is: [WASEpoint], [WASE hashtag] [Program Counter (0 initially)], [Program listing URI], [Unix Timestamp], [Output URI] [,Input URI (optional)] [, Input URI 2 (optional)]

In the case of this computation, the message and computation sequence would look like:

• @engineyard sends: “@ey-sort #wase, 0, bit.ly/7yQK6, 1256850843, bit.ly/2uhGcl, bit.ly/3kl0xs”

…. @ey-sort reads the message from @engineyard in its twitter list and parses the message. First it GETs the program listing from bit.ly/7yQK6, GETS the input data set from bit.ly/3kl0xs, sorts it, PUTS the output to bit.ly/2uhGcl, then looks for the 0+1 WASEpoint in the program listing (@ey-sort) and then..

• @ey-sort sends: “@ey-firsthalf #wase, 1, bit.ly/7yQK6, 1256850875, bit.ly/2uhGcl”

…. @ey-firsthalf reads the message from @ey-sort in its twitter list and parses the message. First it GETs the program listing from bit.ly/7yQK6, GETS the input data set from bit.ly/2uhGcl, halves it, then PUTS the output to bit.ly/2uhGcl,  looks for the 1+1 WASEpoint in the program listing (@ey-firsthalf) and then..

• @ey-firsthalf sends (to itself): “@ey-firsthalf #wase, 2, bit.ly/7yQK6, 1256850885, bit.ly/2uhGcl”

… etc. …

• @ey-firsthalf sends: “@engineyard #wase, 3, bit.ly/7yQK6, 1256850899, bit.ly/2uhGcl”

— finally @engineyard receives this message with the pointer to the final location of output data.

A few new things here. There’s a program counter that tells the WASEpoint where in the program listing the computation is, and there’s a Unix timestamp (could be useful for discarding messages that get held up in the twitterverse?). If no input URI is specified, then the WASEpoint should use the Output URI as both Input and Output locations. One restriction that we will enforce for the contest is that a WASEPOINT MAY NOT DECREMENT A PROGRAM COUNTER: to avoid infinite looping. (Update: And a WASEpoint must conserve the program listing and output URI’s from the input to the output message.)

Hey, maybe we should have some basic error handling. Hmm, let’s say @ey-firsthalf is expecting a standard JSON object but the input data fails to parse properly. Let’s have it send the following message:

• @ey-firsthalf sends: “@engineyard, #wase, -1, bit.ly/7yQK6, 1256850885, bit.ly/2uhGcl

So the error message structure is: [Update: first WASEpoint in program listing], [WASE hashtag], [Negative of Program Counter], [Program listing URI], [Unix Timestamp], [Output URI] [, Input URI (optional)] [, Input URI 2 (optional)]

Note that there are no type declarations in the message format because the only data-type supported by WASE are JSON objects and arrays.

What are Guidelines for the Contest?

Apart from the message format and data guidelines above — here are additional guidelines:

1. Each contestant may register no more than 5 WASE endpoints/twitter accounts. WASEpoints must be registered here to be eligible for use. Your WASEpoints must follow @eycontest. This is also where you should go to pick and choose good WASEpoints for constructing your app. Each contest entry must use a minimum of 10 WASEpoints from at least four separate contestants, where each WASEpoints performs functionally significant data operations. You must supply your own Output URI!

2. Please do not submit WASEpoints whose twitter accounts you do not own :-) We do not want to encourage the business of spamming Ashton Kutcher with mysterious messages. We will test each submitted WASEpoint with a DM to make sure they are legitimate.

3. Source code for your WASEpoint must be posted to a public repository (e.g codaset, github, sourceforge, kenai) for other contestants to inspect :-) If observed behavior deviates from the posted code (aka you have filed a prank WASEpoint), then your entry and all your WASEpoints will be removed from the registered list.

4. A WASEpoint must not store state, and may not rely on any state data other than the input data (of course, it’s easy to generate a private data set programmatically, but this will also be considered state). Trivial WASEpoints (e.g. identity) will be disqualified, although triviality is hard to define, you know it when you see it.

5. You must use bit.ly as your URL shortener (to make everyone’s job building parsers easier—and bit.ly has a http: interface.

7. UPDATE: Challenge calculations submissions must be in the form of a RETWEET to @engineyard of the first message in your WASE program listing from your home twitter account. The final WASEpoint in your program listing should be @eycontest. You must be following @engineyard with your home account in order to enter.

8. We must be able to reproduce your computation using your program listing and our own output URI.

9. We STRONGLY encourage people to write their WASEpoints in Ruby, but we’ll also accept Perl, Scala and Python. Although, be prepared for people avoiding your WASEpoint since the common denominator among people reading this blog is the fact that they know Ruby!

10. We may alter these guidelines along the way, based on your input and feedback, although the spirit and philosophy of them will remain.

However, when you have jobs that use resources from multiple machines, lockrun isn’t adequate. Consider a cron job that builds a local index of a bunch of NFS-mounted files. You don’t want more than one consumer of that NFS volume to index it at a time, otherwise performance will be degraded.

Enter distlockrun: it works similarly to lockrun except that it talks to a central server for mutual exclusion instead of locking a file.

First of all, you need the lock server running. It’s really lightweight, so you can run it on any machine you’ve already got. For example, on Engine Yard Cloud, it should be run on the database master in a screen session.

Start the server like so: distlockrun-server

By default, it starts on port 7890, but that can be changed with the --port option.

Then, on the client, run your job as follows: distlockrun --server {server's hostname} big-expensive-indexer.rb

If there’s already a big-expensive-indexer.rb running, then distlockrun will just exit. Otherwise, it runs the job and then tells the lock server that it’s finished.

That’s all it takes to get cluster-wide mutual exclusion; short but sweet!