codahale.com

The Rest Of The Story

2011-11-29T00:00:00+02:00

About a week ago, I mentioned in passing to an old co-worker on Twitter that Yammer was moving some of our stack from Scala to Java. A few days later, Donald Fischer (the CEO of Typesafe) emailed me at my personal account, asking for more details. He CC'ed Martin Odersky, the lead designer of Scala and Typesafe’s Chief Architect. Given that the two people best-situated to improve Scala had just asked me about my experience over the past two years of using Scala, I wrote a long, considered, brutally honest response.

In the process of composing that email, I asked a few personal friends to review a draft of it. One of those friends shared that draft with someone else, who shared it with someone else, etc. etc. and today I woke up to find my email to Donald and Martin splashed across Hacker News and Twitter. I deleted the gist, but the cat was out of the bag, and now I find myself having to publicly explain the context of a private conversation.

I wrote that email for a very specific reason: Donald asked me for my opinion. If someone asks me for an honest opinion of them or their work, in private, I feel morally compelled to be as honest as I can with them. The only way we can ever know how we appear to other people is through other people; the only way we can know what others think of what we build is to ask and listen to them. The fundamental asymmetry of consciousness means triangulation is the only path to understanding. Anyone running a business understands this, at some level: what customers say does not by itself constitute objective reality, but does offer critical insight into it that one is essentially incapable of discovering by oneself.

But it’s simplistic and naive to assume that I wrote what I did in an unguarded moment and that somehow this represents a more truthful account of what I’d say in public. The most that you can possibly know about this is the text of my email to Donald and Martin, not the context.

Yes, that email is not what I would say in public. The Scala community needs another giant blog post about ways in which someone doesn’t like Scala like I need a hole in my head, and I’d rather suck a dog’s nose dry than lend a hand to the nerd slapfights on Hacker News. The world has yet to take me aside and ask me for my opinion of it, and in the past few years I’ve found that it’s far more profitable to build things rather than tilt at windmills.

So.

Should you use Scala? Is Java better?

(You’re asking the wrong questions.)

(This was posted at codahale.com.)

You Can’t Sacrifice Partition Tolerance

2010-10-07T00:00:00+02:00

I’ve seen a number of distributed databases recently describe themselves as being “CA” —that is, providing both consistency and availability while not providing partition-tolerance. To me, this indicates that the developers of these systems do not understand the the CAP theorem and its implications.

A Quick Refresher

In 2000, Dr. Eric Brewer gave a keynote at the Proceedings of the Annual ACM Symposium on Principles of Distributed Computing¹ in which he laid out his famous CAP Theorem: a shared-data system can have at most two of the three following properties: Consistency, Availability, and tolerance to network Partitions. In 2002, Gilbert and Lynch² converted “Brewer’s conjecture” into a more formal definition with an informal proof. As far as I can tell, it’s been misunderstood ever since.

So let’s be clear on the terms we’re using.

On Consistency

From Gilbert and Lynch²:

Atomic, or linearizable, consistency is the condition expected by most web services today. Under this consistency guarantee, there must exist a total order on all operations such that each operation looks as if it were completed at a single instant. This is equivalent to requiring requests of the distributed shared memory to act as if they were executing on a single node, responding to operations one at a time.

Most people seem to understand this, but it bears repetition: a system is consistent if an update is applied to all relevant nodes at the same logical time. Among other things, this means that standard database replication is not strongly consistent. As anyone whose read replicas have drifted from the master knows, special logic must be introduced to handle replication lag.

That said, consistency which is both instantaneous and global is impossible. The universe simply does not permit it. So the goal here is to push the time resolutions at which the consistency breaks down to a point where we no longer notice it. Just don’t try to act outside your own light cone…

On Availability

Again from Gilbert and Lynch²:

For a distributed system to be continuously available, every request received by a non-failing node in the system must result in a response. That is, any algorithm used by the service must eventually terminate … [When] qualified by the need for partition tolerance, this can be seen as a strong definition of availability: even when severe network failures occur, every request must terminate.

Despite the notion of “100% uptime as much as possible,” there are limits to availability. If you have a single piece of data on five nodes and all five nodes die, that data is gone and any request which required it in order to be processed cannot be handled.

(N.B.: A 500 The Bees They're In My Eyes response does not count as an actual response any more than a network timeout does. A response contains the results of the requested work.)

On Partition Tolerance

Once more, Gilbert and Lynch²:

In order to model partition tolerance, the network will be allowed to lose arbitrarily many messages sent from one node to another. When a network is partitioned, all messages sent from nodes in one component of the partition to nodes in another component are lost. (And any pattern of message loss can be modeled as a temporary partition separating the communicating nodes at the exact instant the message is lost.)

This seems to be the part that most people misunderstand.

Some systems cannot be partitioned. Single-node systems (e.g., a monolithic Oracle server with no replication) are incapable of experiencing a network partition. But practically speaking these are rare; add remote clients to the monolithic Oracle server and you get a distributed system which can experience a network partition (e.g., the Oracle server becomes unavailable).

Network partitions aren’t limited to dropped packets: a crashed server can be thought of as a network partition. The failed node is effectively the only member of its partition component, and thus all messages to it are “lost” (i.e., they are not processed by the node due to its failure). Handling a crashed machine counts as partition-tolerance. (Update: I was wrong about this part. See this for more.) (N.B.: A node which has gone offline is actually the easiest sort of failure to deal with—you’re assured that the dead node is not giving incorrect responses to another component of your system.)

For a distributed (i.e., multi-node) system to not require partition-tolerance it would have to run on a network which is guaranteed to never drop messages (or even deliver them late) and whose nodes are guaranteed to never die. You and I do not work with these types of systems because they don’t exist.

Given A System In Which Failure Is An Option

Michael Stonebraker’s assertions aside, partitions (read: failures) do happen, and the chances that any one of your nodes will fail jumps exponentially as the number of nodes increases:

P(any failure) = 1 – P(individual node not failing)^{number of nodes}

If a single node has a 99.9% chance of not failing in a particular time period, a cluster of 40 has a 96.1% chance not failing. In other words, you’ve got around a 4% chance that something will go wrong. (And this is assuming that your failures are unrelated; in reality, they tend to cascade.)

Therefore, the question you should be asking yourself is:

In the event of failures, which will this system sacrifice? Consistency or availability?

Choosing Consistency Over Availability

If a system chooses to provide Consistency over Availability in the presence of partitions (again, read: failures), it will preserve the guarantees of its atomic reads and writes by refusing to respond to some requests. It may decide to shut down entirely (like the clients of a single-node data store), refuse writes (like Two-Phase Commit), or only respond to reads and writes for pieces of data whose “master” node is inside the partition component (like Membase).

This is perfectly reasonable. There are plenty of things (atomic counters, for one) which are made much easier (or even possible) by strongly consistent systems. They are a perfectly valid type of tool for satisfying a particular set of business requirements.

Choosing Availability Over Consistency

If a system chooses to provide Availability over Consistency in the presence of partitions (all together now: failures), it will respond to all requests, potentially returning stale reads and accepting conflicting writes. These inconsistencies are often resolved via causal ordering mechanisms like vector clocks and application-specific conflict resolution procedures. (Dynamo systems usually offer both of these; Cassandra’s hard-coded Last-Writer-Wins conflict resolution being the main exception.)

Again, this is perfectly reasonable. There are plenty of data models which are amenable to conflict resolution and for which stale reads are acceptable (ironically, many of these data models are in the financial industry) and for which unavailability results in massive bottom-line losses. (Amazon’s shopping cart system is the canonical example of a Dynamo model³).

But Never Both

You cannot, however, choose both consistency and availability in a distributed system.

As a thought experiment, imagine a distributed system which keeps track of a single piece of data using three nodes—A, B, and C—and which claims to be both consistent and available in the face of network partitions. Misfortune strikes, and that system is partitioned into two components: {A,B} and {C}. In this state, a write request arrives at node C to update the single piece of data.

That node only has two options:

Accept the write, knowing that neither A nor B will know about this new data until the partition heals.
Refuse the write, knowing that the client might not be able to contact A or B until the partition heals.

You either choose availability (Door #1) or you choose consistency (Door #2). You cannot choose both.

To claim to do so is claiming either that the system operates on a single node (and is therefore not distributed) or that an update applied to a node in one component of a network partition will also be applied to another node in a different partition component magically.

This is, as you might imagine, rarely true.

A Readjustment In Focus

I think part of the problem with practical interpretations of the CAP theorem, especially with Gilbert and Lynch’s formulation, is the fact that most real distributed systems do not require atomic consistency or perfect availability and will never be called upon to perform on a network suffering from arbitrary message loss. Consistency, Availability, and Partition Tolerance are the Platonic ideals of a distributed system—we can partake of them enough to meet business requirements, but the nature of reality is such that there will always be compromises.

When it comes to designing or evaluating distributed systems, then, I think we should focus less on which two of the three Virtues we like most and more on what compromises a system makes as things go bad. (Because they will go bad.)

This brings us to an earlier bit of Brewer wisdom: yield and harvest, which come from Fox and Brewer’s “Harvest, Yield, and Scalable Tolerant Systems”⁴.

We assume that clients make queries to servers, in which case there are at least two metrics for correct behavior: yield, which is the probability of completing a request, and harvest, which measures the fraction of the data reflected in the response, i.e. the completeness of the answer to the query.

On Yield

In a later article⁵, Brewer expands on yield and its uses:

Numerically, this is typically very close to uptime, but it is more useful in practice because it directly maps to user experience and because it correctly reflects that not all seconds have equal value. Being down for a second when there are no queries has no impact on users or yield, but reduces uptime. Similarly, being down for one second at peak and off-peak times generates the same uptime, but vastly different yields because there might be an order-of-magnitude difference in load between the peak second and the minimum-load second. Thus we focus on yield rather than uptime.

On Harvest

Harvest is a far more overlooked metric, especially in the age of the relational database. If we imagine working on a search engine, however, we can imagine there being separate indexes for each word. The index of web pages which use the word “cute” is on node A, the index of web pages which use the word “baby” is on node B, and the index for the word “animals” is on machine C. A search, then, for “cute baby animals” which combined results from nodes A, B, and C would have a 100% harvest. If nodeB was unavailable, however, we might return a result for just “cute animals,” which would be a harvest of 66%. In other words:

harvest = data available/complete data

Another example would be a system which stores versions of documents. In the event that some nodes are down, the system could choose to present the most recent version of a document that it could find, even if it knew there was a probability that it was not the most recent version it had stored.

A Better Heuristic

Whether a system favors yield or harvest (or is even capable of reducing harvest) tends to be an outcome of its design.

As Brewer puts it⁵:

The key insight is that we can influence whether faults impact yield, harvest, or both. Replicated systems tend to map faults to reduced capacity (and to yield at high utilizations), while partitioned systems tend to map faults to reduced harvest, as parts of the database temporarily disappear, but the capacity in queries per second remains the same.

In terms of general advice to people building distributed systems (and really, who isn’t these days?), I think the following is far more effective:

Despite your best efforts, your system will experience enough faults that it will have to make a choice between reducing yield (i.e., stop answering requests) and reducing harvest (i.e., giving answers based on incomplete data). This decision should be based on business requirements.

Well Now What

It’s incredibly unlikely this will change the way people will build distributed systems—far more of our motivation comes from business concerns (“our shopping carts cannot go down” or “there would be no way for us to reconcile this later”). What I’d like to see, though, is far fewer people unknowingly describing their systems as logical impossibilities.

tl;dr

Of the CAP theorem’s Consistency, Availability, and Partition Tolerance, Partition Tolerance is mandatory in distributed systems. You cannot not choose it. Instead of CAP, you should think about your availability in terms of yield (percent of requests answered successfully) and harvest (percent of required data actually included in the responses) and which of these two your system will sacrifice when failures happen.

References (i.e., Things You Should Read)

Brewer. Towards robust distributed systems. Proceedings of the Annual ACM Symposium on Principles of Distributed Computing (2000) vol. 19 pp. 7-10

Gilbert and Lynch. Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News (2002) vol. 33 (2) pp. 59

DeCandia et al. Dynamo: Amazon’s highly available key-value store. SOSP ‘07: Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles (2007)

Fox and Brewer. Harvest, yield, and scalable tolerant systems. Hot Topics in Operating Systems, 1999. Proceedings of the Seventh Workshop on (1999) pp. 174 – 178

Brewer. Lessons from giant-scale services. Internet Computing, IEEE (2001) vol. 5 (4) pp. 46 – 55

(As a sad postscript: most of the theoretical papers I’ve referenced are about a decade old and all of them are freely available online.)

Updated October 8, 2010

Dr. Brewer approves (somewhat).

Updated October 21, 2010

Dr. Stonebraker does not approve (somehwat).

Updated October 22, 2010

In his response, Dr. Stonebraker says:

In Coda’s view, the dead node is in one partition and the remaining N-1 nodes are in the other one. The guidance from the CAP theorem is that you must choose either A or C, when a network partition is present. As is obvious in the real world, it is possible to achieve both C and A in this failure mode. You simply failover to a replica in a transactionally consistent way. Notably, at least Tandem and Vertica have been doing exactly this for years. Therefore, considering a node failure as a partition results in an obviously inappropriate CAP theorem conclusion.

He is correct: a dead (or wholly partitioned) node can receive no requests, and so the other nodes can easily compensate without compromising consistency or availability here. My error lies in forgetting that Gilbert and Lynch’s formulation of availability requires only non-failing nodes to respond, and that without ≥1 live nodes in a partition there isn’t the option for split-brain syndrome. I regret the error and thank both him and Dr. Brewer for pointing this out.

That said, Dr. Stonebraker’s assertion that “surviving [partitions] will not ‘move the needle’ on availability because higher frequency events will cause global outages” is wrong. Multi-node failures may be rarer than single-node failures, but they are still common enough to have serious effects on business. In my limited experience I’ve dealt with long-lived network partitions in a single data center (DC), PDU failures, switch failures, accidental power cycles of whole racks, whole-DC backbone failures, whole-DC power failures, and a hypoglycemic driver smashing his Ford pickup truck into a DC’s HVAC system. And I’m not even an ops guy.

The fact of the matter is that most real-world systems require substantially less in the way of consistency guarantees than they do in the way of availability guarantees. Amazon’s shopping carts, for example, do not require the full ACID treatment, nor do Twitter’s timelines or Facebook’s news feeds or Google’s indexes. Even the canonical example of an isolated transaction—a transfer of funds between bank accounts—happens with a 24-hour window of indeterminacy. In fact, one of the few financial transactions which actually resembles a database transaction is the physical exchange of cash for goods—a totally analog experience.

But whereas failures of consistency are tolerated or even expected, just about every failure of availability means lost money. Every failed Google search means fewer ads served and advertisers charged; every item a user can’t add to their shopping cart means fewer items sold; every unprocessed credit charge risks a regulatory fine. The choice of availability over consistency is a business choice, not a technical one.

Given this economic context, it becomes clear why most practitioners at any interesting scale meet their business needs using highly-available, eventually consistent systems.

(This was posted at codahale.com.)

How To Safely Store A Password

2010-01-31T00:00:00+02:00

Use `bcrypt`

Use bcrypt. Use bcrypt. Use bcrypt. Use bcrypt. Use bcrypt. Use bcrypt. Use bcrypt. Use bcrypt. Use bcrypt.

Why Not {`MD5`, `SHA1`, `SHA256`, `SHA512`, `SHA-3`, etc}?

These are all general purpose hash functions, designed to calculate a digest of huge amounts of data in as short a time as possible. This means that they are fantastic for ensuring the integrity of data and utterly rubbish for storing passwords.

A modern server can calculate the MD5 hash of about 330MB every second. If your users have passwords which are lowercase, alphanumeric, and 6 characters long, you can try every single possible password of that size in around 40 seconds.

And that’s without investing anything.

If you’re willing to spend about 2,000 USD and a week or two picking up CUDA, you can put together your own little supercomputer cluster which will let you try around 700,000,000 passwords a second. And that rate you’ll be cracking those passwords at the rate of more than one per second.

Salts Will Not Help You

It’s important to note that salts are useless for preventing dictionary attacks or brute force attacks. You can use huge salts or many salts or hand-harvested, shade-grown, organic Himalayan pink salt. It doesn’t affect how fast an attacker can try a candidate password, given the hash and the salt from your database.

Salt or no, if you’re using a general-purpose hash function designed for speed you’re well and truly effed.

`bcrypt` Solves These Problems

How? Basically, it’s slow as hell. It uses a variant of the Blowfish encryption algorithm’s keying schedule, and introduces a work factor, which allows you to determine how expensive the hash function will be. Because of this, bcrypt can keep up with Moore’s law. As computers get faster you can increase the work factor and the hash will get slower.

How much slower is bcrypt than, say, MD5? Depends on the work factor. Using a work factor of 12, bcrypt hashes the password yaaa in about 0.3 seconds on my laptop. MD5, on the other hand, takes less than a microsecond.

So we’re talking about 5 or so orders of magnitude. Instead of cracking a password every 40 seconds, I’d be cracking them every 12 years or so. Your passwords might not need that kind of security and you might need a faster comparison algorithm, but bcrypt allows you to choose your balance of speed and security. Use it.

tl;dr

Use bcrypt.

Updated February 24th, 2011

I’ve been getting pretty regular emails about this article for the past year, and I figured I’d address some of the concerns here rather than have the same conversations over and over again.

Isn’t bcrypt just Blowfish? Where do you store the password?

Please read the Provos & Mazières paper. bcrypt is an adaptive password hashing algorithm which uses the Blowfish keying schedule, not a symmetric encryption algorithm.

You said salts aren’t helpful, but what about rainbow tables? Why would you suggest people not use salts?

As the Provos & Mazières paper describes, bcrypt has salts built-in to prevent rainbow table attacks. So I’m not saying salts are without purpose, I’m saying that they don’t prevent dictionary or brute force attacks (which they don’t).

Rainbow tables, despite their recent popularity as a subject of blog posts, have not aged gracefully. CUDA/OpenCL implementations of password crackers can leverage the massive amount of parallelism available in GPUs, peaking at billions of candidate passwords a second. You can literally test all lowercase, alphabetic passwords which are ≤7 characters in less than 2 seconds. And you can now rent the hardware which makes this possible to the tune of less than $3/hour. For about $300/hour, you could crack around 500,000,000,000 candidate passwords a second.

Given this massive shift in the economics of cryptographic attacks, it simply doesn’t make sense for anyone to waste terabytes of disk space in the hope that their victim didn’t use a salt. It’s a lot easier to just crack the passwords. Even a “good” hashing scheme of SHA256(salt + password) is still completely vulnerable to these cheap and effective attacks, thus the importance of an adaptive hashing algorithm like bcrypt.

(This was posted at codahale.com.)

FOSS and Male Privilege

2009-08-27T00:00:00+02:00

From the comments on the LWN thread, FSF to host a mini-summit on Women in Free Software:

Bruce Perens

Can we say that women aren’t joining because they, as a population rather than as individuals, are not interested? The other alternative would be to say that men in the field had established mechanisms which were astonishingly effective at keeping them out even though they really were interested, and which still stood today.

…

I’d be a lot more comfortable if I heard it from them, and if they explained what the mechanisms were and how they were so effective that even people who were interested were barred from participating with almost total effectiveness. And why this was not so for a number of other fields.

Skud

Bruce, I started using Linux in 1993 with slackware installed off a stack of floppies. I ran X with fvwm and kermit for dialup Internet. I learnt Perl a couple of years later and have worked professionally and full time with Open Source (mostly LAMP stack) since 1996. I’ve done all-nighters and adrenaline-fuelled hacking runs and totally fscked my PC with broken kernel recompiles. I have founded user groups, hosted mailing lists, launched open source projects, etc. I have contributed to major and minor projects all over the damn place; I regularly get email thanking me for writing one of the best known Perl manpages. I have spoken at conferences all over the world. I am well known by certain segments of USENET, IRC, and mailing lists, and geeks all over the world recognise my name when I travel; friends of mine threaten to get tshirts printed saying “Yes, I know Skud” because of this. I have been chewing people’s ears off about why open source/free software is awesome and world-changing since I was 18 years old. And most of the above information is readily available online. About half the first page of Google results (from where I’m sitting right now) for “women in open source” mention me.

Recently, I have also been documenting issues that women face in open source, linking and discussing and synthesising and summarising and KEYNOTING OSCON. (I started doing this a bit in 1998, but stepped back from it for a while, so most of my women-in-open-source work is more recent.)

And then I look at this thread and see that a) “women are just less passionate about open source than men” and b) that nobody seems to believe us when we say there is a problem.

Fuck that. Follow some of those funny little blue underlined words and DO SOME READING.

Bruce Perens

OK, I’m going to guess that you might be Yuwei Lin, and that you are a woman.

Skud

Bruce, I am Kirrily Robert, as a trivial search would have told you, if we hadn’t met in person on multiple occasions.

Bruce Perens

Oh, sorry. I just went by who came up in the first half page of google searches for women in Open Source. Lyn comes up first here.

tl;dr

Bruce Perens demonstrates one of the “established mechanisms which [are] astonishingly effective at keeping [women] out” of the FOSS community. The irony of his inability to recognize the existence of the mechanisms of which he is an active part is lost on him and depresses the rest of us.

(This was posted at codahale.com.)

A Lesson In Timing Attacks (or, Don’t use MessageDigest.isEquals)

2009-08-13T00:00:00+02:00

Timing attacks are pretty horrible from the perspective of someone trying to write a secure cryptosystem. They work against a programmer’s best instincts—don’t do extra work—to give an attacker with access to a Statistics 101 textbook a good solid grip on your application’s guts.

How the hell does that work?

In short, a timing attack uses statistical analysis of how long it takes your application to do something in order to learn something about the data it’s operating on. For HMACs, this means using the amount of time your application takes to compare a given value with a calculated value to learn information about the calculated value.

Take the recent Keyczar vulnerability that Nate Lawson found. He was able to take the fact that Keyczar used a simple break-on-inequality algorithm to compare a candidate HMAC digest with the calculated digest.

This is the offending code in Python:

return self.Sign(msg) == sig_bytes

and in Java:

return Arrays.equals(hmac.doFinal(), sigBytes);

A value which shares no bytes in common with the secret digest will return immediately; a value which shares the first 15 bytes will return 15 compares later. That’s a difference of perhaps microseconds, but given enough attempts—which is usually easy to arrange in web applications (how many of you throttle requests with bad session cookies?)—the random noise becomes a very predictable, normally distributed skew, leaving only the signal.

Well that doesn’t seem that bad

Oh, but it is.

I can choose what message I want to be authenticated—let’s say a session cookie with a specific user ID—and then calculate 256 possible values:

0000000000000000000000000000000000000000
0100000000000000000000000000000000000000
0200000000000000000000000000000000000000
... snip 250 ...
FD00000000000000000000000000000000000000
FE00000000000000000000000000000000000000
FF00000000000000000000000000000000000000

I go through each of these values until I find one— A100000000000000000000000000000000000000—that takes a fraction of a millisecond longer than the others. I now know that the first byte of what the HMAC for that message should be is A1. Repeat the process for the remaining 19 bytes, and all of a sudden I’m logged in as you.

You can’t possibly measure that, can you?

Right about now, most people are thinking about the improbability of measuring the difference between two array comparisons in a web application, given all the routers, parsers, servers, proxies, etc. in the way.

According to Crosby et al.’s Opportunities And Limits Of Remote Timing Attacks:

We have shown that, even though the Internet induces significant timing jitter, we can reliably distinguish remote timing differences as low as 20µs. A LAN environment has lower timing jitter, allowing us to reliably distinguish remote timing differences as small as 100ns (possibly even smaller). These precise timing differences can be distinguished with only hundreds or possibly thousands of measurements.

So. Almost microsecond resolution with hundreds to thousands of measurements. Who wants to bet that network jitter compensation models get worse instead of better?

A worst-case scenario for guessing an HMAC would require 20 × 256 × n measurements, where n is the number of measurements required to pin down a single byte. So—around 5,000,000 requests. You could do that in less than a week at a barely-perceptible 10 req/s.

It’s an attack which takes some planning and analysis, but it’s viable.

Well crap. Now what?

Instead of using a variable-time algorithm for comparing secrets, you should be using constant-time algorithms. Lawson recommends something like the following in Python:

def is_equal(a, b):
    if len(a) != len(b):
        return False

    result = 0
    for x, y in zip(a, b):
        result |= x ^ y
    return result == 0

In Java, that would look like this:

public static boolean isEqual(byte[] a, byte[] b) {
    if (a.length != b.length) {
        return false;
    }

    int result = 0;
    for (int i = 0; i < a.length; i++) {
      result |= a[i] ^ b[i]
    }
    return result == 0;
}

Yay! Problem solved, right?

Oh, if only.

Check out what’s inside of java.security.MessageDigest as recently as Java 6.0 Update 15:

/**
  * Compares two digests for equality. Does a simple byte compare.
  *
  * @param digesta one of the digests to compare.
  *
  * @param digestb the other digest to compare.
  *
  * @return true if the digests are equal, false otherwise.
  */
public static boolean isEqual(byte digesta[], byte digestb[]) {
    if (digesta.length != digestb.length)
        return false;

    for (int i = 0; i < digesta.length; i++) {
        if (digesta[i] != digestb[i]) {
            return false;
        }
    }
    return true;
}

Wait What

Yep. Byte-by-byte comparison; returns on first inequality. Just what we don’t need.

I’ll be blunt here: any Java application which compares client-provided data to a secret value using MessageDigest.isEqual is vulnerable to timing attacks. This includes HMACs, decryption results, etc.

I reported this to Sun on July 22nd, 2009. It’s Bug # 6863503, which isn’t publicly viewable due to the security concerns. Besides the automated email with the ticket number, I haven’t heard anything from Sun since then. In my bug report, I was explicit about my intent to follow through according to the RFPolicy, which says

D. If the MAINTAINER goes beyond 5 working days without any communication to the ORIGINATOR, the ORIGINATOR may choose to disclose the ISSUE. The MAINTAINER is responsible for providing regular status updates (regarding the resolution of the ISSUE) at least once every 5 working days.

So here I am, fully disclosing a rather large cryptographic vulnerability in one of the largest programming platforms there is. I can’t tell if that’s terrible or awesome.

tl;dr

Replace your usage of MessageDigest.isEqual with a constant-time algorithm, like the one above.

Every time you compare two values, ask yourself: what could someone do if they knew either of these values? If the answer is at all meaningful, use a constant-time algorithm to compare them.

A Side Note

This would be substantially less of an issue if more cryptographic libraries had better encapsulation. An HMAC is not just a series of characters or bytes—why treat it as such? Why have such a crucial piece of cryptography squirt out its state for others to man-handle?

Thanks

Nate Lawson’s Keyczar find got me thinking about timing attacks in my own code. His When Crypto Attacks talk should be required watching for everyone with access to a compiler.

Updated August 13, 2009

As sophacles on Hacker News pointed out, I had overly refactored the suggested constant-time algorithms and introduced a more subtle timing attack vulnerability via the return statement’s boolean expression short-circuit. The algorithm has been updated to fix this.

Updated December 3, 2009

The timing attack vulnerability in MessageDigest was fixed in Java SE 6 Update 17.

(This was posted at codahale.com.)

How Not To Fix A Security Bug

2009-06-14T00:00:00+02:00

November 25th, 2008

Tadayoshi Funaba opens Bug #794 in the Ruby Issue Tracking System describing a segmentation fault when huge decimal strings are converted into BigDecimal instances.

November 26th, 2008

Matz closes Bug #794 with r20359 to Ruby 1.9’s trunk.

November 27th, 2008 to June 2nd, 2009

Every site running Ruby 1.8.x which creates BigDecimal instances from client-provided data is vulnerable to a Denial-of-Service attack which Ruby Core developers have already fixed but not backported.

June 3rd, 2009

CVE-2009-1904 is assigned.

June 7th, 2009

ruby-core notifies the vendor-sec mailing list of the vulnerability.

June 8th, 2009

Bug 273213 is created in the Gentoo bug tracker to address CVE-2009-1904. Like most tickets for as-yet-undisclosed security vulnerabilities, the ticket was marked confidential and that “no information should be disclosed until [the vulnerability] is made public.”
Michael Koziarski creates a private GitHub project, bigdecimal-segfault-fix with a workaround for the bug.
Kirk Haines commits a change to the RubySpec project which adds a test to ensure Ruby implementations don’t “segfault when using a very large string to build [a BigDecimal].”

June 9th, 2009

The vulnerability is announced as well as the release of Ruby 1.8.7-p173.
A ticket is added to FreeBSD’s security bug tracker to address CVE-2009-1904.
Michael Koziarski announces the vulnerability to the rails-security mailing list.
Kirk Haines adds Backport #1589 to the Ruby Issue Tracking System describing a patch which “eliminate[s] some BigDecimal bugs.”
Kirk Haines announces the release of Ruby 1.8.6-p369, which includes a fix for CVE-2009-1904.
Michael Koziarski makes the bigdecimal-segfault-fix project public on GitHub.

June 10th, 2009

Barry Hess finds a bug introduced in 1.8.7-p173 breaks BigDecimal#to_f.
A ticket is added to the Red Hat bug tracker to address CVE-2009-1904.
The vulnerability is announced on the Ruby On Rails blog.
A ticket is added to the Ubuntu bug tracker to address CVE-2009-1904.
A ticket is added to the Debian bug tracker to address CVE-2009-1904.

June 12th, 2009

A fix is released for FreeBSD.

June 13th, 2009

A fix is released for Debian Unstable. This fix contains the bug that Barry Hess found.

June 14th, 2009

No fix has been released for Ubuntu.
No fix has been released for Red Hat.
No fix has been released for Fedora Core.
No fix has been released for Gentoo.

June 15th, 2009

Ruby 1.8.7-p174 is released without the BigDecimal#to_f bug.

tl;dr

This is not a coordinated disclosure. This is a clusterfuck. If you are responsible for running a secure MRI/Ruby installation, your only hope is to pay attention to all changes made to Ruby’s trunk and backport any fixes yourself. Depending on your operating system vendor is not a viable strategy, as downstream vendors are not given sufficient advance warning and are presented with fixes which introduce other bugs or do not apply cleanly to the last released version.

Updated June 16th, 2009

Michael Koziarski dropped me a note to clarify a few things. First, his GitHub project was private and was only opened to the public after the vulnerability was announced. Second, ruby-core sent an email to the vendor-sec mailing list 48 hours before the disclosure. I’ve updated the timeline to reflect these changes.
James Ludlow let me know about a bug introduced in 1.8.7-p173 and fixed in 1.8.7-p174.
Updated my conclusions based on the new information. Still not happy, but more accurate in what I’m unhappy about.

(This was posted at codahale.com.)

What Makes Jersey Interesting: Injection Providers

2009-05-16T00:00:00+02:00

Ok, let’s get back to the geeky stuff.

Another interesting thing about Jersey (in addition to parameter classes) is the way it uses dependency injection.

Jersey doesn’t have an abstract base class for resources, like Rails' ActionController::Base or Restlet’s Resource. This removes the obvious way of retrieving information about the incoming request: from the base class.

Take this Rails controller action, for example:

def show
  render(:text => "You asked for #{request.url}")
end

request is a method defined on ActionController::Base which returns an object encapsulating the information about the current HTTP request.

But how would you test this action in isolation? Hopefully the base class provides an easy way of passing in a mock request object, or else you’re stuck modifying instance variables or partially mocking the class you’re trying to test.

So what’s a better way of doing that?

Jersey avoids this situation by injecting the required information into the resource class:

public String show(@Context UriInfo uriInfo) {
  return "You asked for " + uriInfo.getAbsolutePath();
}

Jersey detects the @Context annotation and automatically injects a UriInfo with the current request’s data into the method. You can do this for HttpHeaders, SecurityContext, and a few other Jersey classes.

When it comes time to test this class, it’s a simple matter of making a mock UriInfo and passing it in:

@Test
public void itReturnsTheRequestURI() throws Exception {
    final MyResource resource = new MyResource();
    
    final UriInfo uriInfo = mock(UriInfo.class);
    when(uriInfo.getAbsolutePath()).thenReturn("/wooooo");
    
    assertThat(resource.show(uriInfo), is("You asked for /wooooo"));
}

Because the resource class doesn’t go out and get the UriInfo itself, it’s much easier to test.

What takes this feature from Neat to Indispensable is the fact that Jersey opens this infrastructure to you: it’s easy to write your own injection providers to peel arbitrary bits of an HTTP request off and inject them into your resources.

1.. 2.. 3.. Example time!

As an easy example, let’s take the request’s locale, as determined by its Accept-Language header. Out of the box, Jersey gives us access to this via HttpHeaders#getAcceptableLanguages(), which returns a list of Locale instances in order of preference.

But it’s a tedious thing to have our resource class have an HttpHeaders instance injected and then get the first item from the getAcceptableLanguages() results:

public String uppercase(@Context HttpHeaders headers) {
    return "this is lowercase".toUppercase(headers.getAcceptableLanguages().get(0));
}

In order to test that, we’ll have to come up with an HttpHeaders mock and stub its getAcceptableLanguages() to return a list of locales. Bleagh.

Update: If you’re wondering why you’d need a Locale to convert a string to uppercase, check out the Turkish language. The uppercase version of i (U+0069) is İ (U+0130), and the lowercase version of I (U+0049) is ı (U+0131). It matters.

Now make it uglier

In keeping with my “so what if it looks good on a slide” motif here, I’m also going to throw in error handling: what if the user doesn’t specify a locale?

public String uppercase(@Context HttpHeaders headers) {
    final List<Locale> locales = headers.getAcceptableLanguages();
    final Locale selectedLocale;
    if (locales.isEmpty()) {
        selectedLocale = Locale.US;
    } else {
        selectedLocale = locales.get(0);
    }
    return "this is lowercase".toUppercase(selectedLocale);
}

That’s goddamn horrible—we’ll have to test that logic all over the place, lest we end up throwing an IndexOutOfBoundsException because someone’s HTTP client has funny ideas about valid locales are.

It would be nice to have the locale-selecting code in its own class, and to do that in the same way that our HttpHeaders instance was injected.

I assume you have some kind of plan

In order to do that we’ll need to write two things:

A class implementing Injectable<Locale>, which will be responsible for extracting a Locale from an HTTP request context.
A class implementing InjectableProvider<Locale>, which will be responsible for injecting instances of #1.

Jersey has a specific class to handle the first responsibility: AbstractHttpContextInjectable which is used to inject information from the HTTP context into resource classes. The second responsibility is simple enough to not require a template base class.

Ok, let’s do it

Luckily for us, we can kill two birds with one stone and implement both complimentary responsibilities in a single class:

@Provider
public class LocaleProvider
      extends AbstractHttpContextInjectable<Locale>
      implements InjectableProvider<Context, Type> {
    
    @Override
    public Injectable<E> getInjectable(ComponentContext ic, Context a, Type c) {
        if (c.equals(Locale.class)) {
            return this;
        }

        return null;
    }

    @Override
    public ComponentScope getScope() {
        return ComponentScope.PerRequest;
    }
    
    @Override
    public Locale getValue(HttpContext c) {
        final Locales locales = c.getRequest().getAcceptableLanguages();
        if (locales.isEmpty()) {
          return Locale.US;
        }
        return locales.get(0);
    }
}

This is kind of a complicated class. Let’s cover a few things.

The @Provider annotation marks it so that Jersey will add it as an injection provider.
The getInjectable method checks to see that c is Locale—that is, if Jersey is asking this provider if it can inject a Locale. If so, it returns itself to do the injection. Otherwise, it returns null to indicate there’s nothing it can inject.
The getScope method indicates that the returned injectable is only meaningful on a per-request basis.
The getValue method is where the real logic happens. Once the LocaleProvider instance has been returned from getInjectable, Jersey passes it an HttpContext, which contains all the relevant information about the HTTP request. It returns the most-preferred locale, including error handling.

When we put LocaleProvider in a package that Jersey’s configured to scan, we can reduce our resource class logic to this:

public String uppercase(@Context Locale locale) {
    return "this is lowercase".toUppercase(locale);
}

Then our test looks like this:

@Test
public void itReturnsAnUppercaseString() {
    final MyResource resource = new MyResource();
    
    assertThat(
        resource.uppercase(Locale.CANADA),
        is("THIS IS LOWERCASE")
    );
}

But I don’t think we’re done yet.

How many times you wanna write this thing

Much like parameter classes, our code gets cleaner the more injection providers we write, so we need to extract out the guts into a base class:

public abstract class AbstractInjectableProvider<E>
      extends AbstractHttpContextInjectable<E>
      implements InjectableProvider<Context, Type> {

    private final Type t;

    public AbstractInjectableProvider(Type t) {
        this.t = t;
    }

    @Override
    public Injectable<E> getInjectable(ComponentContext ic, Context a, Type c) {
        if (c.equals(t)) {
            return getInjectable(ic, a);
        }

        return null;
    }

    public Injectable<E> getInjectable(ComponentContext ic, Context a) {
        return this;
    }

    @Override
    public ComponentScope getScope() {
        return ComponentScope.PerRequest;
    }
}

Now our provider is sleek and shiny:

@Provider
public class LocaleProvider extends AbstractInjectableProvider<Locale> {
    public LocaleProvider() {
        super(Locale.class);
    }

    @Override
    public Locale getValue(HttpContext c) {
        final Locales locales = c.getRequest().getAcceptableLanguages();
        if (locales.isEmpty()) {
          return Locale.US;
        }
        return locales.get(0);
    }
}

Now both our resource class and our LocaleProvider are composed and testable.

I love it when a plan comes together.

tl;dr

Jersey has an internal dependency injection system which allows you to write small, focused classes to extract aspects of an HTTP request—in our case, the request’s locale—and inject them into your resource classes as an object of an appropriate type. This makes for smaller, more composed, more testable resource classes, which in turn makes for an application which is easier to change.

(This was posted at codahale.com.)

Ruby and Male Privilege

2009-05-10T00:00:00+02:00

Act One

I read this today:

A female computer science professor wrote:

[A]t a conference in France, a male speaker (French), who was speaking about the importance of testing, showed an overhead slide of a naked woman with a caption of the sort—‘Would you buy this product without testing it first?’ There were only 2 or 3 women in the audience (of about 150), but I had fleeting feelings of having accidentally walked into a stag party and wondering if he had either not expected any women to be there or had discounted the importance of directing his remarks to the women in the audience.

—Ellen Spertus, Why are There so Few Female Computer Scientists?

Dr. Spertus wrote that paper in 1991.

In 1991, I used a gift certificate I won in a junior high car-washing fundraiser content to purchase a tape of Pearl Jam’s Ten from the local record store.

Pearl Jam.

Junior High.

A tape.

Act Two

On April 18th, 2009, Matt Aimonetti gave a presentation on CouchDB entitled CouchDB: Perform Like A Pr0n Star.

You can see the rest of the slides here.

Intermission

In the eighteen intervening years, what’s changed?

Let me rephrase: in the lifetime of this year’s college freshman, what’s changed?

That’s a depressing question.

Here’s another one: when she graduates in 2013, what will have changed for that freshman?

Act Three

A few days ago, I noticed this exchange on Twitter:

And this gem from David Heinemeier Hansson, creator of Rails:

I do think the spheres of the female body gives life wonder. Suppressing all our animal instincts is futile.

Ladies and gentlemen, the leading lights of the Ruby community.

tl;dr

Read Why Are There So Few Female Computer Scientists and HOWTO Encourage Women In Linux. And honestly, I’m beginning to wonder if the women who aren’t joining the Ruby community aren’t making the right choice.

Update: Also be sure to read The Male Programmer Privilege Checklist for examples of the subtle privileges that male programmers enjoy, almost always without knowing it.

(This was posted at codahale.com.)

When Formality Works

2009-05-07T00:00:00+02:00

On his blog, Yehuda Katz writes:

One of the things I love the most about the Ruby community is how easy it is to try out small mutations in practices, which leads to very rapid evolution in best practices. Rather than having the community look toward authority to design, plan, and implement “best practices” (a la the JSR model), members of the Ruby community try different things, and have rapidly made refinements to the practices over time.

In general, I agree with Yehuda about Obie’s current advertising campaign: playing hot-or-not with the bullet points in a HashRocket sales brochure does not make for a compelling discussion of what makes good software, nor does it actually help the community. (But then, I don’t think helping the community is the intent behind RMM—I think it’s about drumming up business by being thoughtleaders.)

But I have an important nit to pick.

That’s not how the JSR model works.

Java programmers don’t sit around feeling helpless waiting for JSR-9918: Doing That Thing You Get Paid To Do to be finalized. Instead, they haul off and implement crazy experiments and solve their problems in new and unique ways just like every other programming community.

In the Java Community Process, you write a JSR proposal describing what you’d like to standardize and why. Here’s the original proposal for JSR-311, about RESTful web services for Java. Then you assemble an expert group—a group of people who are both knowledgeable and interested in the subject—and write a bunch of drafts, go through a bunch of reviews, and finally end up with both a free reference implementation of the standard and a test suite to verify API compliance with the standard.

Horrible, I know.

It is incredibly bureaucratic, yes. But it’s a standardization process. It’s not where innovation starts, it’s where it ends. And that’s as it should be.

The whole point of a standard is to describe a fixed set of practices that people can take for granted. Once they can take it for granted, they can begin to focus on other things.

For example, Rack.

One of the more exciting things to have happened in the Ruby web development community is Rack. It’s not a very sexy project—one can’t claim to be a web framework middleware ninja—but it has an incredible amount of utility: it’s standardized the interface between web application containers—Mongrel, Thin, Ebb, Passenger, Glassfish, Jetty, Tomcat, JBoss, SpringSource, Google App Engine, WEBrick, LiteSpeed, Fuzed, CGI, FastCGI, SCGI, EventedMongrel, SwiftipliedMongrel—and Ruby web applications.

Thanks to Rack’s acceptance, you write a web application to work with a minimal interface and deploy it on a wide variety of infrastructure without needing to write your own adapter code. You no longer need to care about the glue code between your web server and your web application. You can then spend time doing other things, like caring about your actual web application.

Why did Rack succeed?

Rack succeeded because it has a spec, a reference implementation and even a test suite to ensure spec compatibility.

It doesn’t matter if Christian Neukirchen—not that he would—hauls off and destroys Rack as a library; the spec still exists, and implementations of it can be written again and again.

But a spec is not sufficient, obviously. You can’t just crack open Word and start writing fiction in order to make your project gain traction. You need to extract from existing projects a common pattern, round off rough edges, and resolve long-standing issues. Rack did that. It took the various Rails dispatch glue code and other half-assed Ruby middleware implementations, looked at what worked for the Python community, came up with some iterations, got an astounding amount of feedback from concerned, knowledgeable people in the Ruby community, and ended up producing something which has seen widespread adoption in a short period of time.

I’m not sure why Yehuda feels compelled to bring out the scare quotes when referring to these as “best practices.” It’s a good process and it obviously works.

Ruby needs more of this.

The Ruby community has not had a great history of doing this. Ruby Change Requests, a mechanism for proposed changes to the Ruby language, were mothballed because ruby-core simply didn’t want to deal with it. It’s much easier just making changes and having everyone else run around fixing the ways in which the latest patch release of Ruby totally breaks their code. It was discontent with this process which produced the RubySpec project which, say, looked toward authority to design, plan, and implement “best practices” regarding the Ruby language.

Why?

Because trying to build a business on top of a Ruby interpreter in which ruby-core is constantly trying out “small mutations” is really goddamn annoying.

Not everything needs a spec—actually, very few things need a spec. But trying to build the fundamentals of interoperability without a spec and an open process is—like Ruby 1.8.7—bound to fail.

tl;dr

The path to success for a Ruby library and a JSR are the same: pick a problem where diversity of interface poses a problem, extract a common solution, get a bunch of feedback from ~~stakeholders~~interested people in the community who are working in the area, build a solid reference implementation which people can easily use, build a test suite so implementers can red/green their projects, and write a well-defined, simple spec.

It worked for servlets, it worked for Hibernate, it’s working for Restlet, it’s working for Joda Time, it’ll work for Spring and Guice, and it worked for Rack.

(This was posted at codahale.com.)

What Makes Jersey Interesting: Parameter Classes

2009-05-02T00:00:00+02:00

For folks who have known me for a while, this may come as a bit of a shock: these days I’m spending a lot of time working with Java. And I’m having a lot of fun.

lol wut

No really.

This is due in no small part to the fact that I’m working on writing RESTful web services using a really neat framework: Jersey.

What Is This Jersey You Speak Of

Jersey is the reference implementation of JSR311, which is the Java community’s incredibly bureaucratic way of coming up with a decent API for writing RESTful web services. Despite the gray-flannel-suit feel to it, it’s actually a delight to work with.

Broadly speaking, Jersey maps resources to classes, and HTTP verbs to methods.

Here’s an example resource class:

@Path("/helloworld")
@Produces(MediaType.TEXT_PLAIN)
public class HelloWorldResource {
  @GET
  public String sayHello() {
    return "Hello, world!";
  }
}

The @Path annotation marks the class as a resource class and tells Jersey what URIs the resource is responsible for. When a request comes in for /helloworld, Jersey routes that to a HelloWorldResource instance.

The @Produces annotation allows Jersey to perform content negotiation. If a request comes in with a Accept: image/jpeg header, Jersey will respond with a 406 Not Acceptable.

The @GET annotation tells Jersey that the sayHello() method is responsible for handling GET requests. If a resource class doesn’t have a method to handle an HTTP verb, Jersey will respond with a 405 Method Not Allowed.

When a GET request comes in, Jersey calls sayHello(). The String that’s returned gets turned into an HTTP response entity, and you’re off to the races.

There’s a lot more to it, but that’s Jersey and JSR311 in a nutshell.

What this article is about is how a Jersey application handles change—you can find anything about a framework which will look good on a slide but end up sucking horribly in real life (see: Rails' respond_to).

For this article, I’m going to write a weekday calculator. You give it a date, and it tells you what day of the week the day was (or will be) on. Not super-useful, sure, but my boss won’t let me paste huge chunks of our source code here; you’ll have to settle for a contrived example.

Round One: The Simplest Thing Possible

The first thing I’ll do is sketch out a skeleton resource class. Here’s a first swing:

@Path("/v1/weekday/{date}")
@Produces(MediaType.TEXT_PLAIN)
public class SkeletonWeekdayResource {
  @GET
  public String getWeekday(@PathParam("date") String date) {
    return date + " is on a ???.";
  }
}

You’ll notice that the getWeekday method takes an argument, date, which is annotated with @PathParam. The @PathParam annotation pulls the date variable from the resource’s URI template (/v1/weekday/{date}), turns it into a String, and passes it to the getWeekday method.

Here’s a sample request/response:

GET /v1/weekday/20060714 HTTP/1.1
Host: localhost:8080
Accept: */*

And our resource class responds with:

HTTP/1.1 200 OK
Content-Type: text/plain

20060714 is on a ???.

This isn’t much more complicated than HelloWorldResource; we’re still in could-be-crap-but-looks-good-on-a-slide territory. So let’s add the guts of the resource—date parsing and weekday calculation. Because Java’s Calendar and Date classes are hilariously bad, I’m going to use Joda Time, which kicks ass.

Round Two: Now Make It Work

@Path("/v2/weekday/{date}")
@Produces(MediaType.TEXT_PLAIN)
public class NaiveWeekdayResource {
  private static final DateTimeFormatter ISO_BASIC = ISODateTimeFormat.basicDate();
  
  @GET
  public String getWeekday(@PathParam("date") String dateAsString) {
    final DateTime date = ISO_BASIC.parseDateTime(dateAsString);
    return dateAsString + " is on a " + date.dayOfWeek().getAsText() + ".";
  }
}

The changes here are obvious: we use ISO_BASIC, a parser and formatter, to turn dateAsString into a DateTime, date. date.dayOfWeek() returns a property which we turn into text and send back to the client.

Now it does what we want:

GET /v2/weekday/20060714 HTTP/1.1
Host: localhost:8080
Accept: */*

And then:

HTTP/1.1 200 OK
Content-Type: text/plain

20060714 is on a Friday.

But this could still be a Potemkin application. So let’s do something you rarely see in slide shows. Let’s throw some bad input at it.

Round Three: Oh Yeah, Error Handling

What happens when someone asks for an invalid date?

GET /v2/weekday/200607f14 HTTP/1.1
Host: localhost:8080
Accept: */*

Oh geez:

HTTP/1.1 500 Invalid format: "200607f14" is malformed at "f14"
Content-Type: text/html; charset=iso-8859-1

<big-ass stack trace complaining about the date>

That’s not terrible, but it needs to change.

First, 500 Internal Server Erroris the wrong response. The problem isn’t with the server’s state, it’s with the request. A better response would be 400 Bad Request—that way the client knows not to retry the request, and we can add an explanation of what about the request needs to change before it will be acceptable.

Second, unloading a stack trace on random passers-by is bad form. They don’t care, and they probably shouldn’t know what kind of magic is behind the scenes.

So let’s add some error handling:

@Path("/v3/weekday/{date}")
@Produces(MediaType.TEXT_PLAIN)
public class BetterWeekdayResource {
  private static final DateTimeFormatter ISO_BASIC = ISODateTimeFormat.basicDate();
  
  @GET
  public String getWeekday(@PathParam("date") String dateAsString) {
    try {
      final DateTime date = ISO_BASIC.parseDateTime(dateAsString);
      return dateAsString + " is on a " + date.dayOfWeek().getAsText() + ".";
    } catch (IllegalArgumentException e) {
      throw new WebApplicationException(
        Response
          .status(Status.BAD_REQUEST)
          .entity("Couldn't parse date: " + dateAsString + " (" + e.getMessage() + ")")
          .build()
      );
    }
  }
}

This is a pretty simple approach—catch the exception, and throw a WebApplicationException with an HTTP response explaining the problem. Jersey catches the WebApplicationException and sends the attached Response.

Let’s try that again:

GET /v2/weekday/200607f14 HTTP/1.1
Host: localhost:8080
Accept: */*

Yay!

HTTP/1.1 400 Bad Request
Content-Type: text/plain

Couldn't parse date: 200607f14 (Invalid format: "200607f14" is malformed at "f14")

Ok, so our code is now correct and handles errors, but its readability has suffered—for two lines of domain-specific code, we have nine lines of error handling. Ruh-roh. If we continue with this approach, every date parsing resource in the application will have its own error handling, which means a lot of copying and pasting and testing the error handling and bugs, bugs, bugs.

Here’s where Jersey starts to shine—separation of concerns.

Round Four: Time To Clean

The trick here is to stop accepting Strings and start dealing with domain-specific objects. We can do that easily due to the way that Jersey handles the @PathParam annotation.

From the Jersey docs:

The type of the annotated parameter, field or property must either:

…

Be a primitive type.

Have a constructor that accepts a single String argument.

Have a static method named valueOf that accepts a single String argument (see, for example, Integer#valueOf(String)).

So we can just write a class which takes a single String argument, eh?

Like this:

public class SimpleDateParam {
  private static final DateTimeFormatter ISO_BASIC = ISODateTimeFormat.basicDate();
  private final DateTime date;
  private final String originalValue;
  
  public SimpleDateParam(String date) throws WebApplicationException {
    try {
      this.originalValue = date;
      this.date = ISO_BASIC.parseDateTime(date);
    } catch (IllegalArgumentException e) {
      throw new WebApplicationException(
        Response
          .status(Status.BAD_REQUEST)
          .entity("Couldn't parse date: " + date + " (" + e.getMessage() + ")")
          .build()
      );
    }
  }
  
  public DateTime getDate() {
    return date;
  }
  
  public String getOriginalValue() {
    return originalValue;
  }
}

This is a pretty straight-forward class which takes a string, parses it, and either throws a WebApplicationException or returns an object with a DateTime and the original parameter.

We can change our resource class to accept a SimpleDateParam argument instead of a String, which ends up looking like this:

@Path("/v4/weekday/{date}")
@Produces(MediaType.TEXT_PLAIN)
public class AwesomeWeekdayResource {
  @GET
  public String getWeekday(@PathParam("date") SimpleDateParam dateParam) {
    return dateParam.getOriginalValue()
        + " is on a "
        + dateParam.getDate().dayOfWeek().getAsText()
        + ".";
  }
}

Now that’s nice.

In between our first working resource and this one, we’ve done a few things worth noting:

We extracted date parsing and HTTP-specific error handling into a simple, testable, reusable class.
We made our resource classes more testable. Instead of banging Strings together and testing error handling, we can pass in SimpleDateParam stubs test the actual resource logic, safe in the knowledge that a malformed SimpleDateParam cannot exist.
We made our web service a better HTTP citizen. Instead of freaking out with a 500 THE BEES THEY'RE IN MY EYES mystery response, we provide clients and intermediaries with specific, usable information.

But wait! We’re not done yet!

Round Five: And Stay Solved, Damnit

We can safely assume we’ll be writing a lot of these param classes for any given project—in fact, the more of these we write, the cleaner and more testable our resources are.

Think about it—does your web service accept any of the following things?

URIs
Numbers
Enums (e.g., /posts?status=1 ends up being PostStatus.ACTIVE)
Booleans
Timestamps
IDs with a specific format

Duh. Of course it does. Now how many times do want to write that code? Once. So it behooves us to streamline the param-writing process as much as possible.

Thus:

public abstract class AbstractParam<V> {
  private final V value;
  private final String originalParam;
  
  public AbstractParam(String param) throws WebApplicationException {
    this.originalParam = param;
    try {
      this.value = parse(param);
    } catch (Throwable e) {
      throw new WebApplicationException(onError(param, e));
    }
  }
  
  public V getValue() {
    return value;
  }
  
  public String getOriginalParam() {
    return originalParam;
  }
  
  @Override
  public String toString() {
    return value.toString();
  }
  
  protected abstract V parse(String param) throws Throwable;
  
  protected Response onError(String param, Throwable e) {
    return Response
        .status(Status.BAD_REQUEST)
        .entity(getErrorMessage(param, e))
        .build();
  }
  
  protected String getErrorMessage(String param, Throwable e) {
    return "Invalid parameter: " + param + " (" + e.getMessage() + ")";
  }
}

Which means our param class ends up look like this:

public class DateParam extends AbstractParam<DateTime> {
  private static final DateTimeFormatter ISO_BASIC = ISODateTimeFormat.basicDate();
  
  public DateParam(String param) throws WebApplicationException {
    super(param);
  }

  @Override
  protected DateTime parse(String param) throws Throwable {
    return ISO_BASIC.parseDateTime(param);
  }
}

And our resource class looks like this:

@Path("/v5/weekday/{date}")
@Produces(MediaType.TEXT_PLAIN)
public class FinalWeekdayResource {
  @GET
  public String getWeekday(@PathParam("date") DateParam dateParam) {
    return dateParam.getOriginalParam()
        + " is on a "
        + dateParam.getValue().dayOfWeek().getAsText()
        + ".";
  }
}

tl;dr

Jersey’s approach to handling input is graceful in the face of ugly error handling and edge cases, allowing separation of concerns, encapsulation, and reuse. We started out with a simple resource class, added some functionality, added some ugly error handling, then extracted that into a small, composed, testable class. Any other resource class which needs to parse an ISO 8601 basic date? Solved. The end result is testable and readable.

All this despite the fact that it’s in Java.

You can download all the source code for this project here.

(This was posted at codahale.com.)

codahale.com

The Rest Of The Story

You Can’t Sacrifice Partition Tolerance

A Quick Refresher

On Consistency

On Availability

On Partition Tolerance

Given A System In Which Failure Is An Option

Choosing Consistency Over Availability

Choosing Availability Over Consistency

But Never Both

A Readjustment In Focus

On Yield

On Harvest

A Better Heuristic

Well Now What

tl;dr

References (i.e., Things You Should Read)

Updated October 8, 2010

Updated October 21, 2010

Updated October 22, 2010

How To Safely Store A Password

Use bcrypt

Why Not {MD5, SHA1, SHA256, SHA512, SHA-3, etc}?

Salts Will Not Help You

bcrypt Solves These Problems

tl;dr

Updated February 24th, 2011

FOSS and Male Privilege

Bruce Perens

Skud

Bruce Perens

Skud

Bruce Perens

tl;dr

A Lesson In Timing Attacks (or, Don’t use MessageDigest.isEquals)

How the hell does that work?

Well that doesn’t seem that bad

You can’t possibly measure that, can you?

Well crap. Now what?

Yay! Problem solved, right?

Wait What

tl;dr

A Side Note

Thanks

Updated August 13, 2009

Updated December 3, 2009

How Not To Fix A Security Bug

November 25th, 2008

November 26th, 2008

November 27th, 2008 to June 2nd, 2009

June 3rd, 2009

June 7th, 2009

June 8th, 2009

June 9th, 2009

June 10th, 2009

June 12th, 2009

June 13th, 2009

June 14th, 2009

June 15th, 2009

tl;dr

Updated June 16th, 2009

What Makes Jersey Interesting: Injection Providers

So what’s a better way of doing that?

1.. 2.. 3.. Example time!

Now make it uglier

I assume you have some kind of plan

Ok, let’s do it

How many times you wanna write this thing

tl;dr

Ruby and Male Privilege

Act One

Act Two

Intermission

Act Three

tl;dr

When Formality Works

That’s not how the JSR model works.

For example, Rack.

Why did Rack succeed?

Use `bcrypt`

Why Not {`MD5`, `SHA1`, `SHA256`, `SHA512`, `SHA-3`, etc}?

`bcrypt` Solves These Problems