codahale.com

FOSS and Male Privilege

2009-08-27T00:00:00-07:00

From the comments on the LWN thread, FSF to host a mini-summit on Women in Free Software:

Bruce Perens

Can we say that women aren’t joining because they, as a population rather than as individuals, are not interested? The other alternative would be to say that men in the field had established mechanisms which were astonishingly effective at keeping them out even though they really were interested, and which still stood today.

…

I'd be a lot more comfortable if I heard it from them, and if they explained what the mechanisms were and how they were so effective that even people who were interested were barred from participating with almost total effectiveness. And why this was not so for a number of other fields.

Skud

Bruce, I started using Linux in 1993 with slackware installed off a stack of floppies. I ran X with fvwm and kermit for dialup Internet. I learnt Perl a couple of years later and have worked professionally and full time with Open Source (mostly LAMP stack) since 1996. I've done all-nighters and adrenaline-fuelled hacking runs and totally fscked my PC with broken kernel recompiles. I have founded user groups, hosted mailing lists, launched open source projects, etc. I have contributed to major and minor projects all over the damn place; I regularly get email thanking me for writing one of the best known Perl manpages. I have spoken at conferences all over the world. I am well known by certain segments of USENET, IRC, and mailing lists, and geeks all over the world recognise my name when I travel; friends of mine threaten to get tshirts printed saying “Yes, I know Skud” because of this. I have been chewing people’s ears off about why open source/free software is awesome and world-changing since I was 18 years old. And most of the above information is readily available online. About half the first page of Google results (from where I'm sitting right now) for “women in open source” mention me.

Recently, I have also been documenting issues that women face in open source, linking and discussing and synthesising and summarising and KEYNOTING OSCON. (I started doing this a bit in 1998, but stepped back from it for a while, so most of my women-in-open-source work is more recent.)

And then I look at this thread and see that a) “women are just less passionate about open source than men” and b) that nobody seems to believe us when we say there is a problem.

Fuck that. Follow some of those funny little blue underlined words and DO SOME READING.

Bruce Perens

OK, I'm going to guess that you might be Yuwei Lin, and that you are a woman.

Skud

Bruce, I am Kirrily Robert, as a trivial search would have told you, if we hadn’t met in person on multiple occasions.

Bruce Perens

Oh, sorry. I just went by who came up in the first half page of google searches for women in Open Source. Lyn comes up first here.

tl;dr

Bruce Perens demonstrates one of the “established mechanisms which [are] astonishingly effective at keeping [women] out” of the FOSS community. The irony of his inability to recognize the existence of the mechanisms of which he is an active part is lost on him and depresses the rest of us.

(This was posted at codahale.com.)

A Lesson In Timing Attacks (or, Don't use MessageDigest.isEquals)

2009-08-13T00:00:00-07:00

Timing attacks are pretty horrible from the perspective of someone trying to write a secure cryptosystem. They work against a programmer’s best instincts—don’t do extra work—to give an attacker with access to a Statistics 101 textbook a good solid grip on your application’s guts.

How the hell does that work?

In short, a timing attack uses statistical analysis of how long it takes your application to do something in order to learn something about the data it’s operating on. For HMACs, this means using the amount of time your application takes to compare a given value with a calculated value to learn information about the calculated value.

Take the recent Keyczar vulnerability that Nate Lawson found. He was able to take the fact that Keyczar used a simple break-on-inequality algorithm to compare a candidate HMAC digest with the calculated digest.

This is the offending code in Python:

return self.Sign(msg) == sig_bytes

and in Java:

return Arrays.equals(hmac.doFinal(), sigBytes);

A value which shares no bytes in common with the secret digest will return immediately; a value which shares the first 15 bytes will return 15 compares later. That’s a difference of perhaps microseconds, but given enough attempts—which is usually easy to arrange in web applications (how many of you throttle requests with bad session cookies?)—the random noise becomes a very predictable, normally distributed skew, leaving only the signal.

Well that doesn’t seem that bad

Oh, but it is.

I can choose what message I want to be authenticated—let’s say a session cookie with a specific user ID—and then calculate 256 possible values:

0000000000000000000000000000000000000000
0100000000000000000000000000000000000000
0200000000000000000000000000000000000000
... snip 250 ...
FD00000000000000000000000000000000000000
FE00000000000000000000000000000000000000
FF00000000000000000000000000000000000000

I go through each of these values until I find one— A100000000000000000000000000000000000000—that takes a fraction of a millisecond longer than the others. I now know that the first byte of what the HMAC for that message should be is A1. Repeat the process for the remaining 19 bytes, and all of a sudden I'm logged in as you.

You can’t possibly measure that, can you?

Right about now, most people are thinking about the improbability of measuring the difference between two array comparisons in a web application, given all the routers, parsers, servers, proxies, etc. in the way.

According to Crosby et al.’s Opportunities And Limits Of Remote Timing Attacks:

We have shown that, even though the Internet induces significant timing jitter, we can reliably distinguish remote timing differences as low as 20µs. A LAN environment has lower timing jitter, allowing us to reliably distinguish remote timing differences as small as 100ns (possibly even smaller). These precise timing differences can be distinguished with only hundreds or possibly thousands of measurements.

So. Almost microsecond resolution with hundreds to thousands of measurements. Who wants to bet that network jitter compensation models get worse instead of better?

A worst-case scenario for guessing an HMAC would require 20 × 256 × n measurements, where n is the number of measurements required to pin down a single byte. So—around 5,000,000 requests. You could do that in less than a week at a barely-perceptible 10 req/s.

It’s an attack which takes some planning and analysis, but it’s viable.

Well crap. Now what?

Instead of using a variable-time algorithm for comparing secrets, you should be using constant-time algorithms. Lawson recommends something like the following in Python:

def is_equal(a, b):
    if len(a) != len(b):
        return False
    
    result = 0
    for x, y in zip(a, b):
        result |= x ^ y
    return result == 0

In Java, that would look like this:

public static boolean isEqual(byte[] a, byte[] b) {
    if (a.length != b.length) {
        return false;
    }
    
    int result = 0;
    for (int i = 0; i < a.length; i++) {
      result |= a[i] ^ b[i]
    }
    return result == 0;
}

Yay! Problem solved, right?

Oh, if only.

Check out what’s inside of java.security.MessageDigest as recently as Java 6.0 Update 15:

/**
  * Compares two digests for equality. Does a simple byte compare.
  * 
  * @param digesta one of the digests to compare.
  * 
  * @param digestb the other digest to compare.    
  *
  * @return true if the digests are equal, false otherwise.
  */
public static boolean isEqual(byte digesta[], byte digestb[]) {
    if (digesta.length != digestb.length)
        return false;
    
    for (int i = 0; i < digesta.length; i++) {
        if (digesta[i] != digestb[i]) {
            return false;
        }
    }
    return true;
}

Wait What

Yep. Byte-by-byte comparison; returns on first inequality. Just what we don’t need.

I’ll be blunt here: any Java application which compares client-provided data to a secret value using MessageDigest.isEqual is vulnerable to timing attacks. This includes HMACs, decryption results, etc.

I reported this to Sun on July 22nd, 2009. It’s Bug # 6863503, which isn’t publicly viewable due to the security concerns. Besides the automated email with the ticket number, I haven’t heard anything from Sun since then. In my bug report, I was explicit about my intent to follow through according to the RFPolicy, which says

D. If the MAINTAINER goes beyond 5 working days without any communication to the ORIGINATOR, the ORIGINATOR may choose to disclose the ISSUE. The MAINTAINER is responsible for providing regular status updates (regarding the resolution of the ISSUE) at least once every 5 working days.

So here I am, fully disclosing a rather large cryptographic vulnerability in one of the largest programming platforms there is. I can’t tell if that’s terrible or awesome.

tl;dr

Replace your usage of MessageDigest.isEqual with a constant-time algorithm, like the one above.

Every time you compare two values, ask yourself: what could someone do if they knew either of these values? If the answer is at all meaningful, use a constant-time algorithm to compare them.

A Side Note

This would be substantially less of an issue if more cryptographic libraries had better encapsulation. An HMAC is not just a series of characters or bytes—why treat it as such? Why have such a crucial piece of cryptography squirt out its state for others to man-handle?

Thanks

Nate Lawson’s Keyczar find got me thinking about timing attacks in my own code. His When Crypto Attacks talk should be required watching for everyone with access to a compiler.

Updated August 13, 2009

As sophacles on Hacker News pointed out, I had overly refactored the suggested constant-time algorithms and introduced a more subtle timing attack vulnerability via the return statement’s boolean expression short-circuit. The algorithm has been updated to fix this.

(This was posted at codahale.com.)

How Not To Fix A Security Bug

2009-06-14T00:00:00-07:00

November 25th, 2008

Tadayoshi Funaba opens Bug #794 in the Ruby Issue Tracking System describing a segmentation fault when huge decimal strings are converted into BigDecimal instances.

November 26th, 2008

Matz closes Bug #794 with r20359 to Ruby 1.9’s trunk.

November 27th, 2008 to June 2nd, 2009

Every site running Ruby 1.8.x which creates BigDecimal instances from client-provided data is vulnerable to a Denial-of-Service attack which Ruby Core developers have already fixed but not backported.

June 3rd, 2009

CVE-2009-1904 is assigned.

June 7th, 2009

ruby-core notifies the vendor-sec mailing list of the vulnerability.

June 8th, 2009

Bug 273213 is created in the Gentoo bug tracker to address CVE-2009-1904. Like most tickets for as-yet-undisclosed security vulnerabilities, the ticket was marked confidential and that “no information should be disclosed until [the vulnerability] is made public.”
Michael Koziarski creates a private GitHub project, bigdecimal-segfault-fix with a workaround for the bug.
Kirk Haines commits a change to the RubySpec project which adds a test to ensure Ruby implementations don’t “segfault when using a very large string to build [a BigDecimal].”

June 9th, 2009

The vulnerability is announced as well as the release of Ruby 1.8.7-p173.
A ticket is added to FreeBSD’s security bug tracker to address CVE-2009-1904.
Michael Koziarski announces the vulnerability to the rails-security mailing list.
Kirk Haines adds Backport #1589 to the Ruby Issue Tracking System describing a patch which “eliminate[s] some BigDecimal bugs.”
Kirk Haines announces the release of Ruby 1.8.6-p369, which includes a fix for CVE-2009-1904.
Michael Koziarski makes the bigdecimal-segfault-fix project public on GitHub.

June 10th, 2009

Barry Hess finds a bug introduced in 1.8.7-p173 breaks BigDecimal#to_f.
A ticket is added to the Red Hat bug tracker to address CVE-2009-1904.
The vulnerability is announced on the Ruby On Rails blog.
A ticket is added to the Ubuntu bug tracker to address CVE-2009-1904.
A ticket is added to the Debian bug tracker to address CVE-2009-1904.

June 12th, 2009

A fix is released for FreeBSD.

June 13th, 2009

A fix is released for Debian Unstable. This fix contains the bug that Barry Hess found.

June 14th, 2009

No fix has been released for Ubuntu.
No fix has been released for Red Hat.
No fix has been released for Fedora Core.
No fix has been released for Gentoo.

June 15th, 2009

Ruby 1.8.7-p174 is released without the BigDecimal#to_f bug.

tl;dr

This is not a coordinated disclosure. This is a clusterfuck. If you are responsible for running a secure MRI/Ruby installation, your only hope is to pay attention to all changes made to Ruby’s trunk and backport any fixes yourself. Depending on your operating system vendor is not a viable strategy, as downstream vendors are not given sufficient advance warning and are presented with fixes which introduce other bugs or do not apply cleanly to the last released version.

Updated June 16th, 2009

Michael Koziarski dropped me a note to clarify a few things. First, his GitHub project was private and was only opened to the public after the vulnerability was announced. Second, ruby-core sent an email to the vendor-sec mailing list 48 hours before the disclosure. I've updated the timeline to reflect these changes.
James Ludlow let me know about a bug introduced in 1.8.7-p173 and fixed in 1.8.7-p174.
Updated my conclusions based on the new information. Still not happy, but more accurate in what I'm unhappy about.

(This was posted at codahale.com.)

What Makes Jersey Interesting: Injection Providers

2009-05-16T00:00:00-07:00

Ok, let’s get back to the geeky stuff.

Another interesting thing about Jersey (in addition to parameter classes) is the way it uses dependency injection.

Jersey doesn’t have an abstract base class for resources, like Rails' ActionController::Base or Restlet’s Resource. This removes the obvious way of retrieving information about the incoming request: from the base class.

Take this Rails controller action, for example:

def show
  render(:text => "You asked for #{request.url}")
end

request is a method defined on ActionController::Base which returns an object encapsulating the information about the current HTTP request.

But how would you test this action in isolation? Hopefully the base class provides an easy way of passing in a mock request object, or else you’re stuck modifying instance variables or partially mocking the class you’re trying to test.

So what’s a better way of doing that?

Jersey avoids this situation by injecting the required information into the resource class:

public String show(@Context UriInfo uriInfo) {
  return "You asked for " + uriInfo.getAbsolutePath();
}

Jersey detects the @Context annotation and automatically injects a UriInfo with the current request’s data into the method. You can do this for HttpHeaders, SecurityContext, and a few other Jersey classes.

When it comes time to test this class, it’s a simple matter of making a mock UriInfo and passing it in:

@Test
public void itReturnsTheRequestURI() throws Exception {
    final MyResource resource = new MyResource();
    
    final UriInfo uriInfo = mock(UriInfo.class);
    when(uriInfo.getAbsolutePath()).thenReturn("/wooooo");
    
    assertThat(resource.show(uriInfo), is("You asked for /wooooo"));
}

Because the resource class doesn’t go out and get the UriInfo itself, it’s much easier to test.

What takes this feature from Neat to Indispensable is the fact that Jersey opens this infrastructure to you: it’s easy to write your own injection providers to peel arbitrary bits of an HTTP request off and inject them into your resources.

1.. 2.. 3.. Example time!

As an easy example, let’s take the request’s locale, as determined by its Accept-Language header. Out of the box, Jersey gives us access to this via HttpHeaders#getAcceptableLanguages(), which returns a list of Locale instances in order of preference.

But it’s a tedious thing to have our resource class have an HttpHeaders instance injected and then get the first item from the getAcceptableLanguages() results:

public String uppercase(@Context HttpHeaders headers) {
    return "this is lowercase".toUppercase(headers.getAcceptableLanguages().get(0));
}

In order to test that, we’ll have to come up with an HttpHeaders mock and stub its getAcceptableLanguages() to return a list of locales. Bleagh.

Update: If you’re wondering why you'd need a Locale to convert a string to uppercase, check out the Turkish language. The uppercase version of i (U+0069) is İ (U+0130), and the lowercase version of I (U+0049) is ı (U+0131). It matters.

Now make it uglier

In keeping with my “so what if it looks good on a slide” motif here, I'm also going to throw in error handling: what if the user doesn’t specify a locale?

public String uppercase(@Context HttpHeaders headers) {
    final List<Locale> locales = headers.getAcceptableLanguages();
    final Locale selectedLocale;
    if (locales.isEmpty()) {
        selectedLocale = Locale.US;
    } else {
        selectedLocale = locales.get(0);
    }
    return "this is lowercase".toUppercase(selectedLocale);
}

That’s goddamn horrible—we’ll have to test that logic all over the place, lest we end up throwing an IndexOutOfBoundsException because someone’s HTTP client has funny ideas about valid locales are.

It would be nice to have the locale-selecting code in its own class, and to do that in the same way that our HttpHeaders instance was injected.

I assume you have some kind of plan

In order to do that we’ll need to write two things:

A class implementing Injectable<Locale>, which will be responsible for extracting a Locale from an HTTP request context.
A class implementing InjectableProvider<Locale>, which will be responsible for injecting instances of #1.

Jersey has a specific class to handle the first responsibility: AbstractHttpContextInjectable which is used to inject information from the HTTP context into resource classes. The second responsibility is simple enough to not require a template base class.

Ok, let’s do it

Luckily for us, we can kill two birds with one stone and implement both complimentary responsibilities in a single class:

@Provider
public class LocaleProvider
      extends AbstractHttpContextInjectable<Locale>
      implements InjectableProvider<Context, Type> {
    
    @Override
    public Injectable<E> getInjectable(ComponentContext ic, Context a, Type c) {
        if (c.equals(Locale.class)) {
            return this;
        }

        return null;
    }

    @Override
    public ComponentScope getScope() {
        return ComponentScope.PerRequest;
    }
    
    @Override
    public Locale getValue(HttpContext c) {
        final Locales locales = c.getRequest().getAcceptableLanguages();
        if (locales.isEmpty()) {
          return Locale.US;
        }
        return locales.get(0);
    }
}

This is kind of a complicated class. Let’s cover a few things.

The @Provider annotation marks it so that Jersey will add it as an injection provider.
The getInjectable method checks to see that c is Locale—that is, if Jersey is asking this provider if it can inject a Locale. If so, it returns itself to do the injection. Otherwise, it returns null to indicate there’s nothing it can inject.
The getScope method indicates that the returned injectable is only meaningful on a per-request basis.
The getValue method is where the real logic happens. Once the LocaleProvider instance has been returned from getInjectable, Jersey passes it an HttpContext, which contains all the relevant information about the HTTP request. It returns the most-preferred locale, including error handling.

When we put LocaleProvider in a package that Jersey’s configured to scan, we can reduce our resource class logic to this:

public String uppercase(@Context Locale locale) {
    return "this is lowercase".toUppercase(locale);
}

Then our test looks like this:

@Test
public void itReturnsAnUppercaseString() {
    final MyResource resource = new MyResource();
    
    assertThat(
        resource.uppercase(Locale.CANADA),
        is("THIS IS LOWERCASE")
    );
}

But I don’t think we’re done yet.

How many times you wanna write this thing

Much like parameter classes, our code gets cleaner the more injection providers we write, so we need to extract out the guts into a base class:

public abstract class AbstractInjectableProvider<E>
      extends AbstractHttpContextInjectable<E>
      implements InjectableProvider<Context, Type> {

    private final Type t;

    public AbstractInjectableProvider(Type t) {
        this.t = t;
    }

    @Override
    public Injectable<E> getInjectable(ComponentContext ic, Context a, Type c) {
        if (c.equals(t)) {
            return getInjectable(ic, a);
        }

        return null;
    }

    public Injectable<E> getInjectable(ComponentContext ic, Context a) {
        return this;
    }

    @Override
    public ComponentScope getScope() {
        return ComponentScope.PerRequest;
    }
}

Now our provider is sleek and shiny:

@Provider
public class LocaleProvider extends AbstractInjectableProvider<Locale> {
    public LocaleProvider() {
        super(Locale.class);
    }

    @Override
    public Locale getValue(HttpContext c) {
        final Locales locales = c.getRequest().getAcceptableLanguages();
        if (locales.isEmpty()) {
          return Locale.US;
        }
        return locales.get(0);
    }
}

Now both our resource class and our LocaleProvider are composed and testable.

I love it when a plan comes together.

tl;dr

Jersey has an internal dependency injection system which allows you to write small, focused classes to extract aspects of an HTTP request—in our case, the request’s locale—and inject them into your resource classes as an object of an appropriate type. This makes for smaller, more composed, more testable resource classes, which in turn makes for an application which is easier to change.

(This was posted at codahale.com.)

Ruby and Male Privilege

2009-05-10T00:00:00-07:00

Act One

I read this today:

A female computer science professor wrote:

[A]t a conference in France, a male speaker (French), who was speaking about the importance of testing, showed an overhead slide of a naked woman with a caption of the sort—‘Would you buy this product without testing it first?’ There were only 2 or 3 women in the audience (of about 150), but I had fleeting feelings of having accidentally walked into a stag party and wondering if he had either not expected any women to be there or had discounted the importance of directing his remarks to the women in the audience.

—Ellen Spertus, Why are There so Few Female Computer Scientists?

Dr. Spertus wrote that paper in 1991.

In 1991, I used a gift certificate I won in a junior high car-washing fundraiser content to purchase a tape of Pearl Jam’s Ten from the local record store.

Pearl Jam.

Junior High.

A tape.

Act Two

On April 18th, 2009, Matt Aimonetti gave a presentation on CouchDB entitled CouchDB: Perform Like A Pr0n Star.

You can see the rest of the slides here.

Intermission

In the eighteen intervening years, what’s changed?

Let me rephrase: in the lifetime of this year’s college freshman, what’s changed?

That’s a depressing question.

Here’s another one: when she graduates in 2013, what will have changed for that freshman?

Act Three

A few days ago, I noticed this exchange on Twitter:

And this gem from David Heinemeier Hansson, creator of Rails:

I do think the spheres of the female body gives life wonder. Suppressing all our animal instincts is futile.

Ladies and gentlemen, the leading lights of the Ruby community.

tl;dr

Read Why Are There So Few Female Computer Scientists and HOWTO Encourage Women In Linux. And honestly, I'm beginning to wonder if the women who aren’t joining the Ruby community aren’t making the right choice.

Update: Also be sure to read The Male Programmer Privilege Checklist for examples of the subtle privileges that male programmers enjoy, almost always without knowing it.

(This was posted at codahale.com.)

When Formality Works

2009-05-07T00:00:00-07:00

On his blog, Yehuda Katz writes:

One of the things I love the most about the Ruby community is how easy it is to try out small mutations in practices, which leads to very rapid evolution in best practices. Rather than having the community look toward authority to design, plan, and implement “best practices” (a la the JSR model), members of the Ruby community try different things, and have rapidly made refinements to the practices over time.

In general, I agree with Yehuda about Obie’s current advertising campaign: playing hot-or-not with the bullet points in a HashRocket sales brochure does not make for a compelling discussion of what makes good software, nor does it actually help the community. (But then, I don’t think helping the community is the intent behind RMM—I think it’s about drumming up business by being thoughtleaders.)

But I have an important nit to pick.

That’s not how the JSR model works.

Java programmers don’t sit around feeling helpless waiting for JSR-9918: Doing That Thing You Get Paid To Do to be finalized. Instead, they haul off and implement crazy experiments and solve their problems in new and unique ways just like every other programming community.

In the Java Community Process, you write a JSR proposal describing what you'd like to standardize and why. Here’s the original proposal for JSR-311, about RESTful web services for Java. Then you assemble an expert group—a group of people who are both knowledgeable and interested in the subject—and write a bunch of drafts, go through a bunch of reviews, and finally end up with both a free reference implementation of the standard and a test suite to verify API compliance with the standard.

Horrible, I know.

It is incredibly bureaucratic, yes. But it’s a standardization process. It’s not where innovation starts, it’s where it ends. And that’s as it should be.

The whole point of a standard is to describe a fixed set of practices that people can take for granted. Once they can take it for granted, they can begin to focus on other things.

For example, Rack.

One of the more exciting things to have happened in the Ruby web development community is Rack. It’s not a very sexy project — one can’t claim to be a web framework middleware ninja—but it has an incredible amount of utility: it’s standardized the interface between web application containers—Mongrel, Thin, Ebb, Passenger, Glassfish, Jetty, Tomcat, JBoss, SpringSource, Google App Engine, WEBrick, LiteSpeed, Fuzed, CGI, FastCGI, SCGI, EventedMongrel, SwiftipliedMongrel — and Ruby web applications.

Thanks to Rack’s acceptance, you write a web application to work with a minimal interface and deploy it on a wide variety of infrastructure without needing to write your own adapter code. You no longer need to care about the glue code between your web server and your web application. You can then spend time doing other things, like caring about your actual web application.

Why did Rack succeed?

Rack succeeded because it has a spec, a reference implementation and even a test suite to ensure spec compatibility.

It doesn’t matter if Christian Neukirchen—not that he would—hauls off and destroys Rack as a library; the spec still exists, and implementations of it can be written again and again.

But a spec is not sufficient, obviously. You can’t just crack open Word and start writing fiction in order to make your project gain traction. You need to extract from existing projects a common pattern, round off rough edges, and resolve long-standing issues. Rack did that. It took the various Rails dispatch glue code and other half-assed Ruby middleware implementations, looked at what worked for the Python community, came up with some iterations, got an astounding amount of feedback from concerned, knowledgeable people in the Ruby community, and ended up producing something which has seen widespread adoption in a short period of time.

I'm not sure why Yehuda feels compelled to bring out the scare quotes when referring to these as “best practices.” It’s a good process and it obviously works.

Ruby needs more of this.

The Ruby community has not had a great history of doing this. Ruby Change Requests, a mechanism for proposed changes to the Ruby language, were mothballed because ruby-core simply didn’t want to deal with it. It’s much easier just making changes and having everyone else run around fixing the ways in which the latest patch release of Ruby totally breaks their code. It was discontent with this process which produced the RubySpec project which, say, looked toward authority to design, plan, and implement “best practices” regarding the Ruby language.

Why?

Because trying to build a business on top of a Ruby interpreter in which ruby-core is constantly trying out “small mutations” is really goddamn annoying.

Not everything needs a spec—actually, very few things need a spec. But trying to build the fundamentals of interoperability without a spec and an open process is—like Ruby 1.8.7—bound to fail.

tl;dr

The path to success for a Ruby library and a JSR are the same: pick a problem where diversity of interface poses a problem, extract a common solution, get a bunch of feedback from ~~stakeholders~~interested people in the community who are working in the area, build a solid reference implementation which people can easily use, build a test suite so implementers can red/green their projects, and write a well-defined, simple spec.

It worked for servlets, it worked for Hibernate, it’s working for Restlet, it’s working for Joda Time, it’ll work for Spring and Guice, and it worked for Rack.

(This was posted at codahale.com.)

What Makes Jersey Interesting: Parameter Classes

2009-05-02T00:00:00-07:00

For folks who have known me for a while, this may come as a bit of a shock: these days I'm spending a lot of time working with Java. And I'm having a lot of fun.

lol wut

No really.

This is due in no small part to the fact that I'm working on writing RESTful web services using a really neat framework: Jersey.

What Is This Jersey You Speak Of

Jersey is the reference implementation of JSR311, which is the Java community’s incredibly bureaucratic way of coming up with a decent API for writing RESTful web services. Despite the gray-flannel-suit feel to it, it’s actually a delight to work with.

Broadly speaking, Jersey maps resources to classes, and HTTP verbs to methods.

Here’s an example resource class:

@Path("/helloworld")
@Produces(MediaType.TEXT_PLAIN)
public class HelloWorldResource {
  @GET
  public String sayHello() {
    return "Hello, world!";
  }
}

The @Path annotation marks the class as a resource class and tells Jersey what URIs the resource is responsible for. When a request comes in for /helloworld, Jersey routes that to a HelloWorldResource instance.

The @Produces annotation allows Jersey to perform content negotiation. If a request comes in with a Accept: image/jpeg header, Jersey will respond with a 406 Not Acceptable.

The @GET annotation tells Jersey that the sayHello() method is responsible for handling GET requests. If a resource class doesn’t have a method to handle an HTTP verb, Jersey will respond with a 405 Method Not Allowed.

When a GET request comes in, Jersey calls sayHello(). The String that’s returned gets turned into an HTTP response entity, and you’re off to the races.

There’s a lot more to it, but that’s Jersey and JSR311 in a nutshell.

What this article is about is how a Jersey application handles change—you can find anything about a framework which will look good on a slide but end up sucking horribly in real life (see: Rails' respond_to).

For this article, I'm going to write a weekday calculator. You give it a date, and it tells you what day of the week the day was (or will be) on. Not super-useful, sure, but my boss won’t let me paste huge chunks of our source code here; you’ll have to settle for a contrived example.

Round One: The Simplest Thing Possible

The first thing I’ll do is sketch out a skeleton resource class. Here’s a first swing:

@Path("/v1/weekday/{date}")
@Produces(MediaType.TEXT_PLAIN)
public class SkeletonWeekdayResource {
  @GET
  public String getWeekday(@PathParam("date") String date) {
    return date + " is on a ???.";
  }
}

You’ll notice that the getWeekday method takes an argument, date, which is annotated with @PathParam. The @PathParam annotation pulls the date variable from the resource’s URI template (/v1/weekday/{date}), turns it into a String, and passes it to the getWeekday method.

Here’s a sample request/response:

GET /v1/weekday/20060714 HTTP/1.1
Host: localhost:8080
Accept: */*

And our resource class responds with:

HTTP/1.1 200 OK
Content-Type: text/plain

20060714 is on a ???.

This isn’t much more complicated than HelloWorldResource; we’re still in could-be-crap-but-looks-good-on-a-slice territory. So let’s add the guts of the resource—date parsing and weekday calculation. Because Java’s Calendar and Date classes are hilariously bad, I'm going to use Joda Time, which kicks ass.

Round Two: Now Make It Work

@Path("/v2/weekday/{date}")
@Produces(MediaType.TEXT_PLAIN)
public class NaiveWeekdayResource {
  private static final DateTimeFormatter ISO_BASIC = ISODateTimeFormat.basicDate();
  
  @GET
  public String getWeekday(@PathParam("date") String dateAsString) {
    final DateTime date = ISO_BASIC.parseDateTime(dateAsString);
    return dateAsString + " is on a " + date.dayOfWeek().getAsText() + ".";
  }
}

The changes here are obvious: we use ISO_BASIC, a parser and formatter, to turn dateAsString into a DateTime, date. date.dayOfWeek() returns a property which we turn into text and send back to the client.

Now it does what we want:

GET /v2/weekday/20060714 HTTP/1.1
Host: localhost:8080
Accept: */*

And then:

HTTP/1.1 200 OK
Content-Type: text/plain

20060714 is on a Friday.

But this could still be a Potemkin application. So let’s do something you rarely see in slide shows. Let’s throw some bad input at it.

Round Three: Oh Yeah, Error Handling

What happens when someone asks for an invalid date?

GET /v2/weekday/200607f14 HTTP/1.1
Host: localhost:8080
Accept: */*

Oh geez:

HTTP/1.1 500 Invalid format: "200607f14" is malformed at "f14"
Content-Type: text/html; charset=iso-8859-1

<big-ass stack trace complaining about the date>

That’s not terrible, but it needs to change.

First, 500 Internal Server Erroris the wrong response. The problem isn’t with the server’s state, it’s with the request. A better response would be 400 Bad Request—that way the client knows not to retry the request, and we can add an explanation of what about the request needs to change before it will be acceptable.

Second, unloading a stack trace on random passers-by is bad form. They don’t care, and they probably shouldn’t know what kind of magic is behind the scenes.

So let’s add some error handling:

@Path("/v3/weekday/{date}")
@Produces(MediaType.TEXT_PLAIN)
public class BetterWeekdayResource {
  private static final DateTimeFormatter ISO_BASIC = ISODateTimeFormat.basicDate();
  
  @GET
  public String getWeekday(@PathParam("date") String dateAsString) {
    try {
      final DateTime date = ISO_BASIC.parseDateTime(dateAsString);
      return dateAsString + " is on a " + date.dayOfWeek().getAsText() + ".";
    } catch (IllegalArgumentException e) {
      throw new WebApplicationException(
        Response
          .status(Status.BAD_REQUEST)
          .entity("Couldn't parse date: " + dateAsString + " (" + e.getMessage() + ")")
          .build()
      );
    }
  }
}

This is a pretty simple approach—catch the exception, and throw a WebApplicationException with an HTTP response explaining the problem. Jersey catches the WebApplicationException and sends the attached Response.

Let’s try that again:

GET /v2/weekday/200607f14 HTTP/1.1
Host: localhost:8080
Accept: */*

Yay!

HTTP/1.1 400 Bad Request
Content-Type: text/plain

Couldn't parse date: 200607f14 (Invalid format: "200607f14" is malformed at "f14")

Ok, so our code is now correct and handles errors, but its readability has suffered—for two lines of domain-specific code, we have nine lines of error handling. Ruh-roh. If we continue with this approach, every date parsing resource in the application will have its own error handling, which means a lot of copying and pasting and testing the error handling and bugs, bugs, bugs.

Here’s where Jersey starts to shine—separation of concerns.

Round Four: Time To Clean

The trick here is to stop accepting Strings and start dealing with domain-specific objects. We can do that easily due to the way that Jersey handles the @PathParam annotation.

From the Jersey docs:

The type of the annotated parameter, field or property must either:

…

Be a primitive type.

Have a constructor that accepts a single String argument.

Have a static method named valueOf that accepts a single String argument (see, for example, Integer#valueOf(String)).

So we can just write a class which takes a single String argument, eh?

Like this:

public class SimpleDateParam {
  private static final DateTimeFormatter ISO_BASIC = ISODateTimeFormat.basicDate();
  private final DateTime date;
  private final String originalValue;
  
  public SimpleDateParam(String date) throws WebApplicationException {
    try {
      this.originalValue = date;
      this.date = ISO_BASIC.parseDateTime(date);
    } catch (IllegalArgumentException e) {
      throw new WebApplicationException(
        Response
          .status(Status.BAD_REQUEST)
          .entity("Couldn't parse date: " + date + " (" + e.getMessage() + ")")
          .build()
      );
    }
  }
  
  public DateTime getDate() {
    return date;
  }
  
  public String getOriginalValue() {
    return originalValue;
  }
}

This is a pretty straight-forward class which takes a string, parses it, and either throws a WebApplicationException or returns an object with a DateTime and the original parameter.

We can change our resource class to accept a SimpleDateParam argument instead of a String, which ends up looking like this:

@Path("/v4/weekday/{date}")
@Produces(MediaType.TEXT_PLAIN)
public class AwesomeWeekdayResource {
  @GET
  public String getWeekday(@PathParam("date") SimpleDateParam dateParam) {
    return dateParam.getOriginalValue()
        + " is on a "
        + dateParam.getDate().dayOfWeek().getAsText()
        + ".";
  }
}

Now that’s nice.

In between our first working resource and this one, we've done a few things worth noting:

We extracted date parsing and HTTP-specific error handling into a simple, testable, reusable class.
We made our resource classes more testable. Instead of banging Strings together and testing error handling, we can pass in SimpleDateParam stubs test the actual resource logic, safe in the knowledge that a malformed SimpleDateParam cannot exist.
We made our web service a better HTTP citizen. Instead of freaking out with a 500 THE BEES THEY'RE IN MY EYES mystery response, we provide clients and intermediaries with specific, usable information.

But wait! We’re not done yet!

Round Five: And Stay Solved, Damnit

We can safely assume we’ll be writing a lot of these param classes for any given project—in fact, the more of these we write, the cleaner and more testable our resources are.

Think about it—does your web service accept any of the following things?

URIs
Numbers
Enums (e.g., /posts?status=1 ends up being PostStatus.ACTIVE)
Booleans
Timestamps
IDs with a specific format

Duh. Of course it does. Now how many times do want to write that code? Once. So it behooves us to streamline the param-writing process as much as possible.

Thus:

public abstract class AbstractParam<V> {
  private final V value;
  private final String originalParam;
  
  public AbstractParam(String param) throws WebApplicationException {
    this.originalParam = param;
    try {
      this.value = parse(param);
    } catch (Throwable e) {
      throw new WebApplicationException(onError(param, e));
    }
  }
  
  public V getValue() {
    return value;
  }
  
  public String getOriginalParam() {
    return originalParam;
  }
  
  @Override
  public String toString() {
    return value.toString();
  }
  
  protected abstract V parse(String param) throws Throwable;
  
  protected Response onError(String param, Throwable e) {
    return Response
        .status(Status.BAD_REQUEST)
        .entity(getErrorMessage(param, e))
        .build();
  }
  
  protected String getErrorMessage(String param, Throwable e) {
    return "Invalid parameter: " + param + " (" + e.getMessage() + ")";
  }
}

Which means our param class ends up look like this:

public class DateParam extends AbstractParam<DateTime> {
  private static final DateTimeFormatter ISO_BASIC = ISODateTimeFormat.basicDate();
  
  public DateParam(String param) throws WebApplicationException {
    super(param);
  }

  @Override
  protected DateTime parse(String param) throws Throwable {
    return ISO_BASIC.parseDateTime(param);
  }
}

And our resource class looks like this:

@Path("/v5/weekday/{date}")
@Produces(MediaType.TEXT_PLAIN)
public class FinalWeekdayResource {
  @GET
  public String getWeekday(@PathParam("date") DateParam dateParam) {
    return dateParam.getOriginalParam()
        + " is on a "
        + dateParam.getValue().dayOfWeek().getAsText()
        + ".";
  }
}

tl;dr

Jersey’s approach to handling input is graceful in the face of ugly error handling and edge cases, allowing separation of concerns, encapsulation, and reuse. We started out with a simple resource class, added some functionality, added some ugly error handling, then extracted that into a small, composed, testable class. Any other resource class which needs to parse an ISO 8601 basic date? Solved. The end result is testable and readable.

All this despite the fact that it’s in Java.

You can download all the source code for this project here.

(This was posted at codahale.com.)