Confessions from a Massive, Nightmarish Data Project

Back in 2000, I went to France to build a genomics platform. A biotech hired me to combine their in-house genome data with that of public repositories like Genbank. The problem was the repositories, all with millions of records, each had their own format. It sounded like a massive, nightmarish data interoperability project. And an ideal fit for a hot new technology : XML.

So I dove in, spending my days designing DTDs, writing parsers, tweaking tags (”taxon” or “species”? attribute or element?). At night I dreamt in ontologies. It was perfect.

Then reality struck. The pipeline was slow: Oracle loaded XML at a crawl. And it was a memory hog, since XSLT required putting full document trees in RAM.

We had a deadline to meet (and, mon dieu, a 35 hour work-week). So we changed course. We hacked our Perl scripts to emit a flat tab-delimited format — “TabML” — which was bulk loaded into Oracle. It wasn’t elegant, but it was fast and it worked.

Yet looking back, I realize that XML was the wrong format from the start. And as I’ll argue here, our unhealthy obsession with XML formats threatens to slow or impede many open data projects, including initiatives like Data.gov.

In the next sections, I discuss how XML fails for Big Data because of its unnatural form, bulk, and complexity. Finally, I generalize to three rules that advocate a more liberal approach to data.

Three Reasons Why XML Fails for Big Data

I. XML Spawns Data Bureaucracy

In its natural habitat, data lives in relational databases or as data structures in programs. The common import and export formats of these environments do not resemble XML, so much effort is dedicated to making XML fit. When more time is spent on inter-converting data — serializing, parsing,translating — than in using it, you’ve created a data bureaucracy.

Indeed, it was what Doug Crockford called “impedance mismatch inefficiencies” that sparked him to create JSON - standardizing Javascript’s object notation as a portable data container.

II. Yes, Size Matters for Data

Size matters for data in a way it does not for documents. Documents are intended for human consumption and have human-sized upper bounds (a lifetime’s worth of reading fits on a thumb drive). Data designed for machine consumption is bounded only by bandwidth and storage.

XML’s expansiveness — for even when compressed, the genie must be let out the bottle at some point — imposes memory, storage, and CPU costs.

III. Complexity Carries a Cost

I never fail to sigh when I open a data file and discover an army of tags, several ranks deep, surrounding the data I need. XML’s complexity imposes costs without commensurate benefits, specifically:

• In-line, element-by-element tagging is redundant. Far preferable is stating the data model separately, and using a lightweight delimiter (such as a comma or a tab).
• Text tags are purported to be self-documenting, but textual meaning is a slippery thing: it’s rare that one can be sure of a tag’s data type without consulting its DTD (in a separate document).
• End-tags support nested structures (such as an aside (within (an aside)). But to facilitate data exchange, flattened out structures are preferable, and arbitrary levels of nesting are best using sparingly.

XML’s complexity inflicts misery on both sides of the data divide: on the publishing side, developers struggle to comply with the latest edicts of a fussy standards group. While data suitors labor to quickly unravel that XML format into something they can use.

Three Rules for XML Rebels

I. Stop Inventing New Formats (as Tim Bray said in 2006)

Before you call for “an XML format for X”, let me tell you a story about LaTeX and MathML. (And while these are document formats, there’s a lesson here for data).

The LaTeX typesetting system is the lingua franca for composing scientific documents. As the one-million plus LaTeX-formatted articles on arXiv.org attest, it is spoken by scientists worldwide.

MathML, on the other hand, is a markup language for mathematics recommended by the W3C. If you’re a scientist looking to use MathML, you have two choices: (i) find a program to convert LaTeX, which you already know, to MathML 3.0 or (ii) familiarize yourself with this handy 354-page spec and code it yourself.

Two years ago, Mike Adams thought of a third way: why not just let people use LaTeX directly in WordPress? So he wrote a plug-in that did it. The applause was deafening.

Spoken languages are strengthened by usage, not by imperial fiat, and data formats are no different. Far better to evolve and adapt the standards we already have (as JSON and SQLite’s file format do), than to fabricate new ones from whole cloth. As John Udell says, “good-enough solutions [that are] here now, and familiar to people, often trump great solutions that aren’t here and wouldn’t be familiar if they were.”

II. Obey The Fifteen Minute Rule

Interviewed several years ago, James Clark stated “If a technology is too complicated, no matter how wonderful it is and how easy it makes a user’s life, it won’t be adopted on a wide scale.”

Accordingly, if you absolutely must develop a new API, language, or format, it should satisfy a simple rule: a person of reasonable ability should be able to get from zero to ‘Hello World’ in fifteen minutes. (This does not preclude complex languages or formats, per se: it does require that additional complexity not be sui generis, but built on some existing foundation, for example.)

Despite a noble vision for the semantic web , the barriers for adopting the W3C’s proposals for linked data are too high. The beauty of original HTML standard was that it was dead simple. The flaw of RDF is that it is too hard.

III. Embrace Lazy Data Modeling

To keep data bureaucracy to a minimum, several Big Data thinkers have advocated a more catholic approach to data: building data stores that accommodate a broad range of data types and formats.

Lazy data modeling is similar to lazy evaluation. The right schema for data depends on future use cases, in as-yet-undeveloped applications. Instead of trying to guess the future, we can store the data “as-is” — and deal with its transformation when (and if) a necessary use case arises. As Michael Franklin and colleagues note: ”the most scarce resource available for semantic integration is human attention.”

This liberal view also reduces barriers for data sharing, barriers which threaten initiatives like Data.gov. The US Census Bureau shouldn’t expend resources to publish in XML if they have a good-enough format available right now.

For the data geeks in the trenches, who are building the next generation of data services, the laws of economics hold fast: there are unlimited opportunities in the face of one limited resource, time. (Which also explains why data geeks seem to get no sleep).

XML’s unfulfilled promise for data testifies that formats can create friction. The easier it is for data to be shared and consumed, the more quickly we’ll realize our visions for smarter businesses and better governments.

(25-Aug-2009 Update: Read a response from open gov advocates at Sunlight Labs).

As others have noted, we’ve reached a tipping point in history: more data is being manufactured by machines — servers, cell phones, GPS-enabled cars — than by people. The early, document-centric web was populated by hand-coded hypertext files; today, a hand-coded web page is as rare as hand-woven clothing.

Through web frameworks, wikis, and blogs, we have industrialized the creation of hypertext. Similarly, we’ve also industrialized the collection of data, and spliced out the human steps in many data flows, such that data entry clerks may soon be as rare as typesetters.

The web we experience will continue to be dominated by documents — e-mail, blogs, and news. And while many sites are data-centric — Google maps, Weather.com, and Yahoo finance — it’s the web that we can’t see that surging with data. It’s not about us, it’s about servers in the cloud mediating entire pipelines of data, only occasionally surfacing in a browser.

But the web’s data architecture is fractious and in flux: many competing standards exist for serializing, parsing, and describing data. As we build out the data web, we ought to embrace standards that mirror data’s form in its natural habitats — as programmatic data structures, relational tables, or key-value pairs — while taking advantage of data’s stream-like nature. Mark-up languages like HTML and XML are ideal for documents, but they are poor containers for data, especially Big Data.

Sacred “Words & Enthusiasm” vs Meaningless Utterances

Documents and data are different.  The table below reflects my thin grasp of the fissure lines, as a step towards arguing why we ought to design around them.

Documents are made of “words and enthusiasm”: sonnets, cake recipes, blog posts, Supreme Court rulings, and dictionary definitions. Their core stuffing is text. Their structure is unpredictable and irregular — even fractal.

Data are not created but collected (something given, not something made): city temperatures, stock prices, web visitors, and home runs. They are observations in time and space, with periodic and predictable structure. Data are reorderable and divisible: you can relay city temperatures in any order, but you can’t rearrange a Shakespearian sonnet without muddling its meaning. Some documents are so meaningful as to be considered sacred.

Data are, in this regard, meaningless on their own; they do not signify, they simply are. These data are the utterances of the spimes that surround us.

Documents as Trees, Data as Streams

The argument for shifting away from markup languages as data formats is not just practical, it’s philosophical: it’s about pivoting our conception away from the dominant metaphor of documents — trees — towards one far more suitable for data — streams.

Trees are rooted and finite: you can’t chop up a tree and easily put it back together again (while XML has made concessions to document fragments, it is not a natural fit).

Streams can be split, sampled, and filtered. The divisibility of data streams lends itself to parallelism in a way that document trees do not. The stream paradigm conceives of data as extending infinitely forward in time. The Twitter data stream has no end: it ought have no end tag.

Conceiving of data as streams moves us out of the realm of static objects and into the realm of signal processing. This is the domain of the living: where the web is not an archive but an organism, reacting in real-time.

XML Considered Harmful for Data

XML is a poor language for data because it solves the wrong problems — those of documents — while leaving many of data’s unique issues unaddressed.   But many promising alternatives exist — microformats like JSON, Thrift, and even SQLite’s file format – as I will detail in a my next post.

In prepping for tonite’s talk at the Google IO Ignite event, this quote inspired me to muse about how sex appeal and statistics might go together: so I chose to mash up a few scatter plots with Andy Warhol’s Marilyn Monroe.

Statisticians’ sex appeal has little to do with their lascivious leanings (ahem, BedPost), and more with the scarcity of their skills.  I believe that the folks to whom Hal Varian is referring are not statisticians in the narrow sense, but rather people who possess skills in three key, yet independent areas:  statistics, data munging, and data visualization.  (In parentheses next to each, I’ve put the salient character trait needed to acquire it).

Skill #1: Statistics (Studying). Statistics is perhaps the most important skill and the hardest to learn. It’s a deep and rigorous discipline, and one that is actively progressing (the widely used method of Least Angle Regression was only recently developed in 2004). I expect to be on its learning curve my entire life. This being the case, people who possess a solid grasp of modern statistics are rare.   And yet problems that require its application continue to multiply.  The text that I was exposed to in graduate school and find to be an unparalleled survey is Hastie, Tibshirani, and Friedman’s Elements of Statistical Learning.

Skill #2: Data Munging (Suffering). The second critical skill mentioned above is “data munging.” Among data geek circles (you can find us with a Twitter search for #rstats), this refers to the painful process of cleaning, parsing, and proofing one’s data before it’s suitable for analysis. Real world data is messy. At best it’s inconsistently delimited or packed into an unnecessarily complex XML schema. At worst, it’s a series of scraped HTML pages or a thoroughly undocumented fixed-width format.

A good data munger excels at turning coffee into regular expressions and parsers, implemented in a high-level scripting language of choice (often Perl, Python, even Javascript). This is problem solving with programming, and quite different from statistics. An aspiration towards elegance — in the form of a perfect XSLT filter, for example — is rarely rewarded, and often punished. A decade ago, I thought that the world’s data would soon be well-structured, and my talent for syntactical incantations of regular expressions would be a moot skill. I was wrong. (Perhaps there’s an analogy with the paper industry: the growing volume of data means we’ll likely need more regular expressions before we need less).

Related to munging but certainly far less painful is the ability to retrieve, slice, and dice well-structured data from persistent data stores, using a combination of SQL, scripting languages (especially Python and its SciPy and NumPy libraries), and even several oldie-but-goodie Unix utilities (cut, join).

And when data sets grow too large to manage on a single desktop, the samurai of data geeks are capable of parallelizing storage and computation with tools like 96-nodes of Postgres, snow and RMPI, Hadoop and Mapreduce, and on Amazon EC2 to boot.

Skill #3: Visualization (Storytelling). This third and last skill that Professor Varian refers to is the easiest to believe one has.  Most of us have had exposure to basic chart-making widgets of Excel (and to date myself, tools like Harvard Graphics). But a little knowledge is a dangerous thing: these software tools are often insufficient when faced with the visualization of large, multivariate data sets.

Here it’s worth making a distinction between two breeds of data visualizations, which differ in their audience and their goals. The first are exploratory data visualizations (as named by John Tukey), intended to faciliate a data analyst’s understanding of the data. These may consist of scatter plot matrices and histograms, where labels and colors are minimally set by default. Their goal is to help develop a hypothesis about the data, and their audience typically numbers one or a small team.

A second kind of data visualization are those intended to communicate to a wider audience, whose goal is to visually advocate for a hypothesis. While most data geeks are facile with exploratory graphics, the ability to create this second kind of visualization, these visual narratives, is again a separate skill — with separate tools.  (R is excellent for static visualizations, but cannot compete with the kinds of rich interactive visualizations that tools like Processing and Flare make possible). Luckily, successful collaboration often occurs between data analysts and designers, the occasional fracas notwithstanding.

The ability to visualize and communicate data is critical, because even with good data and rigorous statistical techniques, if the results of an analysis are poorly visualized, they will not convince: whether it’s an academic discovery or a business proposal.

Put All Three Skills Together: Sexy. Thus with the Age of Data upon us, those who can model, munge, and visually communicate data — call us statisticians or data geeks — are a hot commodity.  I grew up before the age of geek chic, when the computer wizzes were social pariahs, and feature-length movies were dedicated to nerds seeking revenge.  But in the last decade, Steve Jobs became an icon, the Internet became cool, and an entire generation of tech kids grew up well adjusted.  They even built the social web to prove it.   I believe the same could happen to statistics and data geeks too.

(Update Aug-2009:  If you liked this post, consider voting for it at the 2010 SXSW Conference).

• Stamen: Reaching through Maps
• Protovis: A Declarative, Open Source Graphical Toolkit
• A Mathematician’s View: A Visualization is a Hypothesis
• UUorld: Multidimensional Extrusion Maps

Stamen: Reaching through Maps

Eric Rodenbeck (Stamen) started by highlighting several mapping visualizations that Stamen has been hacking on recently and in the past, including Cabspotting in San Francisco , Crimespotting in Oakland, and Olympic Stadium spotting in London.

Eric showed how Stamen has attempted to move away from what Schuyler Erle has dubbed “red dot fever”, whereby the overlayed data can overwhelm our visual attention, and toward allowing various data layers to “reach through” the maps.

For example, the London Olympic maps provide a mixture of schematic, satellite, and webcam images.  These various drill-downs of detail are not all exposed, but rather collaged.  Even more interesting was a movable ‘lens’ that, as it is moved over regions of a map, reveals another layer (reminiscent of a polarized-light based mural at Boston’s MoS).  In these ways, additional layers of data are only selectively brought into focus (echoing a design pattern in Japanese gardening, mie gakure, meaning “seen and unseen”).

One practical gem that Mike Migurski shared regarding the Oakland Crimespotting site was, “the design of a comments section is a huge part of how its perceived and used.”  Nota bene, social web developers.

Protovis: A Declarative, Open Source Graphical Toolkit

Mike Bostock (Stanford CS) introduced Protovis, an extensible visualization toolkit implemented using Javascript’s canvas element. Protovis draws inspiration from Leland Wilkinson’s Grammar of Graphics, which argues for moving away from the prevailing method of building visualizations, where data are simply poured into one of several chart types — pie, stacked bar, or scatter.

Wilkinson argues that visualizations should not be cast from chart typologies, but rather composed of graphical primitives. In Protovis, these primitives include dots, areas, lines, and labels (called “marks”).

Among Protovis’s strengths are:

A More Declarative Syntax for Creating Graphics

One disadvantage of directly using Javascript’s canvas is its imperative style. To draw a diagonal line, the code must manipulate and move a pen using x,y coordinates. With Protovis, however, the code declares (roughly) “add a bar to this graph” (example). Thus Protovis provides a grammar for statements about graphical marks, rather than statements about graphical mechanics.

Visible Open Source

With Protovis, the source code is not just open and available, it’s viewable from within the browser. I have an admittedly personal bias for open source data visualization, but lowering the barriers to sharing source code ultimately drive faster adoption and iteration of visualization techniques.

Mike has used Protovis to recreate classic data visualizations by Will Burtin, Florence Nightingale, William Playfair, and others. You can find these at the Protovis site and in their InfoVis ‘09 paper.

(For those interested in a Wilkinson-inspired approach for graphics in R, check out Hadley Wickham’s ggplot).

A Mathematician’s View: A Visualization is a Hypothesis

Jason Morton (Stanford Mathematics) made the argument that a data visualization is not merely a descriptive vessel, it is a predictive model.

A visualization is a model is because, especially with large data sets, not every dimension of every observation can be shown. Quite simply, a (compressed) 100k data visualization cannot losslessly describe a (compressed) 10 Mb data set: information must be discarded. What remains is a model of the original data, albeit a visual model.

Moreover, a data visualization’s model is predictive: it presents a hypothesis about how observable data points were generated, and implies predictions about future, as-yet-unobserved data.

Seen from this perspective, Stamen’s Crimespotting maps are powerful precisely because they make compelling hypotheses about when and where crime occurs in Oakland. Their London Olympic maps, which integrate time series photographs of the stadium site, take a position about the pace of construction and how it is impacting the landscape.

“Form Ever Follows Function”

And if the function of a data visualization is to make hypotheses, then its form should follow this function. The arbitrary use of color, position, shape, and ornament — only adds noise.

The ever popular Wordle provides a visual model for word distribution in a text: more frequent words are larger. However, a word’s color, position, and font are arbitrarily chosen - they carry no meaning, and model nothing. Indeed, the “randomize” button is an admission of as much (for it does not randomize size).

Adding arbitrary marks or dimensions to a visualization carries two related risks: first, it can obscure the true model that’s trying to be conveyed (what do same-colored have in common?); second, this added complexity, beyond polluting the information channel, has a cost: the visualization is larger. Bar graphs with iPhone ads in the background cannot be succinctly rendered.

The parallels to the modernist movement in architecture are obvious. Adolf Loos wrote in 1908 that “the evolution of culture marches with the elimination of ornament from useful objects.” The American modernist Louis Sullivan proclaimed that “form ever follows function.”

But the truth is that stripping visualizations down to their bare models can be counterproductive. Call it noise or ornamentation, but even visual marks that do not advance a hypothesis can act to support it,  by guiding the eye, providing context, or otherwise speeding the absorption of a pattern by the human brain. At the very least, this functionalist perspective can help data visualizers use ornamentation intentionally, not inadvertently.

UUorld: Multidimensional Extrusion Maps

Zach Wilson (UUorld) showcased his company’s software that simplifies creating and exploring extrusion maps. Among the several interesting applications of his software, Zach showed off a temporal visualization of the spread of swine flu in the United States over the past several weeks.

In response to the critique that layering data dimensions on two-dimensional maps could be done more effectively by use other indicators such as color — instead of the simulation of a third dimension of height — Zach indicated that research has shown that physical dimensions (or their simulation) possess greater visual saliency to the human eye.

Zach also mentioned UUorld’s data portal which contains thousands of downloadable statistics from a variety of public sources; some of which have been used to generate UUorld visualizations.

“Avoiding catastrophe becomes the first principle in bringing color to information: Above all, do no harm.”  — Envisioning Information, Edward Tufte, Graphics Press, 1990   

Color is one of the most abused and neglected tools in data visualization. It is abused when we make poor color choices; it is neglected when we rely on poor software defaults. Yet despite its historically poor treatment at the hands of engineers and end-users alike, if used wisely, color is unrivaled as a visualization tool.

Most of us think twice before walking outside in fluorescent red underoos. If only we were as cautious in choosing colors for infographics. The difference is that few of us design our own clothes. But until good palettes (like ColorBrewer) are commonplace, to get colors that fit our purposes, we must be our own tailors.

While obsessing about how to implement color on the Dataspora Labs’ PitchFX viewer I began with a basic motivating question:

Why use color in data graphics?

If our data are simple, a single color is sufficient, even preferable. For example, below is a scatter plot of 287 pitches thrown by the major league pitcher Oscar Villarreal in 2008. With just two dimensions of data to describe — the x and y location in the strike zone — black and white is sufficient. In fact, this scatter plot is a perfectly lossless representation of the data set (assuming no data points perfectly overlap).

Fig 1. Location of Pitches (Villarreal, HOU, 2008)

But what if we’d like to know more: for instance, what kinds of pitches (curveballs, fastballs) landed where? Or their speed?  Visualizations live in two dimensions, but the world they describe is rarely so confined.

The defining challenge of data visualization is projecting high dimensional data onto a low dimensional canvas. (As a rule, one should never do the reverse: visualize more dimensions than what already exist in the data).

Getting back to our pitching example, if we want to layer another dimension of data — pitch type — into our plot, we have several methods at our disposal:

1. plotting symbols - vary the glyphs that we use (circles, triangles, etc.),
2. small multiples - vary extra dimensions in space, creating a series of smaller plots
3. color - we can color our data, encoding extra dimensions inside a color space

Which techniques you employ depend on the nature of the data and the media of your canvas. I will describe all three by way of example.

Multivariate Method I:  Vary Your Plotting Symbols

Fig 2. Location and Pitch Type (Villarreal, HOU, 2008)

In this plot, I’ve layered the categorical dimension of pitch type into our plot by using four different plotting symbols.

I consider this visualization an abject failure.  In fact, the prize for my most despised graphs in graduate school goes to bacterial growth curves rendered this way . The reason these graphs make our heads hurt is because (i) distinguishing glyphs demands extra attention (versus what academics call ‘pre-attentively processed‘ cues like color), (ii) even after we visually decode the symbols, we have yet another step: mapping symbols to their semantic categories. (Admittedly this can be improved with Chernoff faces or other iconic symbols, where the categorical mapping is self-evident).

Multivariate Method II:  Small Multiples on a Canvas

Folding additional dimensions into a partitioned canvas has a distinguished pedigree in information graphics. It has been employed everywhere from Galileo sunspot illustrations to William Cleveland’s trellis plots. And as Scott Mccloud’s unexpected tour de force on comics makes clear, panels of pictures possess a narrative power that a single, undivided canvas lacks.

In this plot below, the four types of pitches that Oscar throws are splintered horizontally.   By reducing our plot sizes, we’ve given up some resolution in positional information. But in return, patterns that were invisible in our first plot, and obscured in our second (by varied symbols) are now made clear (Oscar throws his fastballs low, but his sliders high).

Fig 3:  Location and Pitch Type (Villarreal, HOU, 2008)

Multiplying plots in space works especially well on printed media, which can hold more than ten times as many dots per square inch as a screen. Both columns and rows can be used to lattice over additional dimensions, the result being a matrix of scatter plots (in R, see the ‘splom‘ function).

Multivariate Method III: Color Your Data

So why bother with color?

First, as compared to most print media, computer displays have fewer units of space, but a broader color gamut. So color is a compensatory strength.

For multi-dimensional data, color can convey additional dimensions inside a unit of space — and can do so instantly. Color differences can be detected within 200 ms, before you’re even conscious of paying attention (the ‘pre-attentive’ concept I mentioned earlier).

But the most important reason to use color in multivariate graphics is that color is itself multidimensional. Our perceptual color space — however you slice it — is three-dimensioned.

In the example below, I’ve used color as a means of encoding a fourth dimension of our pitching data: the speed of pitches thrown. The palette I’ve chosen is a divergent palette that moves along one dimension (think of it as the ‘redness-blueness’ dimension) in the CIELUV color space, while maintaining a constant level of luminosity.

Fig 4. Location, Pitch Type, and Velocity (Villarreal, HOU, 2008)

Holding luminosity constant is important, because luminosity (similar to brightness) determines a color’s visual impact. Bright colors pop, and dark colors recede. A color ramp that varies luminosity along with hue will highlight data points as an artifact of color choice.

I chose only seven gradations of color, so I’m downsampling (in a lossy way) our speed data - but further segmentation of our color ramp is not likely to be perceptible.

I’ve also chosen to use filled circles as my plotting symbol, as opposed to the open circles in all my previous plots. This is done to improve the perception of each pitch’s speed via its color: small patches of color are less perceptible. But a consequence of this choice — compounded by our choice to work with a series of smaller plots — is that more points overlap. We’ve further degraded some of our positional information. However, in our last step, we attempt to recover some of this.

Now I’ve finally brought color to bear on this visualization, but I’ve only encoded a single dimension — speed. Which leads to another question:

If color is three-dimensional, can I encode three dimensions with it?

In theory, yes. Colin Ware researched this exact question. In practice, it’s difficult. It turns out that asking observers to assess the amount of ‘redness’, ‘blueness’, and ‘greenness’ of points is possible, but not intuitive (I suspect it’s somewhat like parsing symbols).

Another complicating factor is that a nontrivial fraction of the population has some form of color blindness. This effectively reduces their color perception to two dimensions.

And finally, the truth is that our sensation of color is not equal along all dimensions; it’s thought the closely related ‘red’ and ‘green’ receptors emerged via duplication of the single long wavelength receptor (useful for detecting ripe from unripe fruits, according to one just-so story).

Because the high level of dichromacy in the population, and because of the challenge of encoding three dimensions in color, I  feel color is best used to encode no more than two dimensions of data.

So, for my last example of our pitching plot data, I will introduce luminosity as a means of encoding the local density of points (using a kernel density estimator). This allows us to recover some of the data lost by increasing the sizes of our plotting symbols.

Fig 5. Location, Pitch Type, Velocity, and Density (Villarreal, HOU, 2008)

Here we have effectively employed a two-dimensional color palette, with blueness-redness varying along one axis for speed, and luminosity varying in the other to denote local density.

One final point about using luminosity. Observing colors in a data visualization involves overloading, in the programming sense. We rely on cognitive functions that were developed for one purpose (perceiving lions) and use them for another (perceiving lines).

Since we can overload color any way we want, whenever possible, we should choose mappings that are natural. Mapping pitch density to luminosity feels right because the darker shadows in our pitch plots imply depth. Likewise, when sampling from the color space, we might as well choose colors found in nature. These are the palettes our eyes were gazing at for the millions of years before #FF0000 showed up.

Color, used thoughtfully and responsibly, can be an incredibly valuable tool in visualizing high dimensional data.

FutureMan Asks: What about Animation?

This discussion has focused on using static graphics in general, and color in particular, as a means of visualizing multivariate data. I’ve purposely neglected one very powerful tool:  motion. The ability to animate graphics multiplies by several orders of magnitude the amount of information that can be packed into a visualization.  But packing  information into a time-varying data structure has to be done by someone (you or me) and from my view, this remains a significant challenge.  Canonical forms of animated visualizations (equivalent to the histograms, box plots, and scatterplots of the static world) are still a ways off, but frameworks like Processing and Prefuse are a promising start towards their development.

Methods

The final product of these five-dimensional pitch plots — for all available data for the 2008 season — can be explored via the PitchFX Django-driven web tool at Dataspora labs.

All of the visualizations here were developed using R and the Lattice graphics package.  (Of note, Hadley Wickham is developing ggplot2, a bold re-write of the R graphics system based on a grammar of graphics).

References for Further Reading

• Ross Ihaka - Lectures on Information Visualization, Lectures 12-14

• Colin Ware - Information Visualization, Ch. 4

• Edward Tufte, Envisioning Information, Ch. 4.

• Deepayan Sarkar - Lattice: Multivariate Data Visualization with R (web site with code)

• Maureen Stone - StoneSoup Consulting (color consultant to Tableau Software)

• Stephen Few, Information Dashboard Design, Ch. 4

I was going to blog all about it — but Tom Carden of Stamen Design already has a great write-up.

… Dataspora invited a few people to a Dataviz Salon yesterday evening. Mike and I went along and huddled in a brick-built basement in SoMa to listen to the following:

.

The panel comprised of four recognized R users from industry:

• Bo Cowgill, Google
• Itamar Rosenn, Facebook
• David Smith, Revolution Computing
• Jim Porzak, The Generations Network (and Co-Chair of our R Users Group)

The panelists were asked to explain how they use R for predictive analytics within their firms, its strengths and weaknesses as a tool, and provide a case study. What follows is my summary with comments.

Panel Introduction

I began by describing R as a programming language with strengths in three areas: (i) data manipulation, (ii) statistics, and (iii) data visualization.

What sets it apart from other data analysis tools? It was developed by statisticians, it’s free software, and it is extensible via user-developed packages — there are nearly 2000 of them as of today at the Comprehensive R Archive Network or CRAN.

Many of these packages can be used for predictive analytics. Jim highlighted Max Kuhn’s caret package , which provides a wrapper for accessing dozens of classification and regression models, from neural networks to naive Bayes.

Bo Cowgill, Google

R is the most popular statistical package at Google, according to Bo Cowgill, and indeed Google is a donor to the R Foundation. He remarked that “The best thing about R is that it was developed by statisticians. The worst thing about R is that… it was developed by statisticians.” Nonetheless, he’s optimistic to see that as the R developer community has expanded, R’s documentation has improved, and its performance has gained.

One theme that Bo first brought up, but which was echoed by others, was that while Google uses R for data exploration and model prototyping, it is not typically used in production: in Bo’s group, R is typically run in a desktop environment.

The typical workflow that Bo thus described for using R was: (i) pulling data with some external tool, (ii) loading it into R, (iii) performing analysis and modeling within R, (iv) implementing a resulting model in Python or C++ for a production environment.

Itamar Rosenn, Facebook

Itamar conveyed how Facebook’s Data Team used R in 2007 to answer two questions about new users: (i) which data points predict whether a user will stay? and (ii) if they stay, which data points predict how active they’ll be after three months?

For the first question, Itamar’s team used recursive partitioning (via the rpart package) to infer that just two data points are significantly predictive of whether a user remains on Facebook: (i) having more than one session as a new user, and (ii) entering basic profile information.

For the second question, they fit the data to a logistic model using a least angle regression approach (via the lars package), and found that activity at three months was predicted by variables related to three classes of behavior: (i) how often a user was reached out to by others, (ii) frequency of third party application use, and (iii) what Itamar termed “receptiveness” — related to how forthcoming a user was on the site.

David Smith, Revolution Computing

David’s firm, Revolution Computing, not only uses R, but R is their core business. David said that “we are to R what Red Hat is to Linux”. His firm addresses some of the pain points of using R, such as (i) supporting older versions of the software and (ii) providing parallel computing in R through their ParallelR suite.

David showcased how one of their life sciences clients used R to classify genomic data through use of the randomForest package, and how the analysis of classification trees could be easily parallelized using their ‘foreach’ package.

He also mentioned that several firms they have worked with do use R in production environments, whereby a particular script is exposed on a server, and a client calls it with some data to return a result (several ways exist to set up R in a client-server manner, such as RServe , rapache , and Biocep).

David evangelizes and educates about R at the Revolutions blog .

Jim Porzak, The Generations Network

Jim (also co-chairs the R Users Group), gave a brief overview of his PAW talk on using R for marketing analytics. In particular, Jim has used the flexclust package to cluster customer survey data for Sun Microsystems, and apply the resulting profiles to identify high-value sales leads.

During the Q & A session, the panelists were asked several questions.

How do you work around R’s memory limitations? (R workspaces are stored in RAM, and thus their size is limited)

Three responses were given (including one from the audience):

(i) use R’s database connectivity (e.g. RMySQL), and pull in only slices of your data, (ii) downsample your data (do you really a billion data points to test your model?), or (iii) run your scripts on a RAM-obsessed colleague’s machine or fire up an virtual server on Amazon’s compute cloud — for up to 15 Gigs.

What’s the general ramp-up process for groups wanting to use R?

Itamar and Bo both indicated that within their groups, almost everyone arrived having learned R in their university studies. Jim Porzak led an R tutorial within his last firm using an internal slide deck.

How easy is it for developers who are not statisticians to learn R?

The consensus seemed to be that R is a difficult language to achieve competency in, vis-a-vis Python, Perl, or other high-level scripting languages.   Jim emphasized, however, that he is a not a statistician - nor were any of our panelists. (As a non-statistician R user myself, I will say this — a consequence of learning R is an improved grasp of statistics. Knowing statistics is a necessary pre-requisite for understanding R’s features, from its data types to its modeling syntax).

How well does R interface with other tools and languages?

There are several packages on CRAN for importing and exporting data to and from Matlab ( RMatlab), Splus, SAS, Excel and other tools.  In addition, there are interfaces for running R within Python ( RPy ) and Java ( RJava ).

The panelists mentioned that they typically run R within a GUIs, either RCommander or Rattle . (Aside: I run R exclusively in emacs using ESS — incidentally, one of its authors was panelist David Smith).

A video of the event is now available courtesy of Ron Fredericks and LectureMaker.

(5/18/09 update - included an overdue reference to linked data!) 

Stuart Kauffman, in one of his books about complexity, discusses tipping points in networks — what he calls a phase transitions — by way of buttons. Suppose you’re sitting on a floor strewn with 400 buttons, and you begin tying them together with pieces of string at random. At first, you have just pairs of buttons.   Then, you have clusters of threes, which in turn get tied into ever larger clumps. The question is: How long until picking any button off the floor pulls them all off together, in one connected mass?

It turns out that this supercluster of buttons doesn’t build gradually as we tie more threads, it emerges suddenly.  This rapid phase transition, from relatively unconnected to mostly connected, occurs right around where we have about half as many threads as buttons (see figure).  This is the tipping point of the system:  where a few threads make a big difference.

A similar phase transition has already occurred with regards to data inside business ecosystems. For the past several decades, an increasing number of business processes– from sales, customer service, shipping - have come online, along with the data they throw off.  As these individual databases are linked, via common formats or labels, a tipping point is reached: suddenly, every part of the company organism is connected to the data center.  And every action — sales lead, mouse click, and shipping update  — is stored.  The result:  organizations are overwhelmed by what feels like a tsunami of data.

The same trend is occurring in the larger universe of data that these organizations inhabit.  Big Data unleashed by the “Industrial Revolution of Data”, whether from public agencies, non-profit institutes, or forward-thinking private firms.

At present, much of the world’s Big Data is iceberg-like: frozen and mostly underwater. It’s frozen because format and meta-data standards make it hard to flow from one place to another:  comparing the SEC’s financial data with that of Europe’s requires common formats and labels (ahem, XBRL) that don’t yet exist. Data is “underwater” when, whether reasons of competitiveness, privacy, or sheer incompetence it’s not shared: US medical records may contain a wealth of data, but much of it is on paper and offline (not so in Europe, enabling studies with huge cohorts).

Yet there’s a slow thaw underway as evidenced by a number of initiatives:  Aaron Swartz’s theinfo.org, Flip Kromer’s infochimps, Carl Malamud’s bulk.resource.org, the Tim-Berners-Lee-inspired LinkedData.org, as well as Numbrary, Swivel, Freebase, and Amazon’s public data sets.  These are all ambitious projects, but the challenge of weaving these data sets together is still greater.

How far are we from the tipping point of Big Data? When will the world’s icebergs of data melt into one sea? More importantly, when it happens, will we be ready to do something useful with it all?

(Aside:  I confess that relating the economic crisis to Darwin is a symptom of an academic bad habit:  namely, mapping every phenomenon onto the intellectual giant of one’s field.  Somewhere there is a psychologist blogging about Freud and the economy).

When does natural selection act?   This question motivates two modern naturalists, Peter and Rosemary Grant, who studied Darwin’s finches over several decades on the Galapagos islands, and whose work is chronicled in Jonathan Weiner’s The Beak of the Finch.

During the wet seasons, it was hard to see how a finch’s beak made any difference to its fitness.

“[F]inches with long thin beaks and short fat parrot-like beaks [were] all hopping on the same lava, eating identical bird food… All those beaks were cracking the same birdseed.” [p.52]

A long line of ornithologists had concluded that the beak of the finch was unimportant.

But despite this, Peter and Rosemary Grant kept returning to the islands, and kept measuring beaks. In 1977, the rainy season brought no rain. Weiner describes what the naturalists’ witnessed:

“They found fewer than two hundred finches alive on the island. Just one finch in seven had made it through the drought… The average beak before the drought was 10.68 millimeters long and 9.42 deep. The average beak of the fortis that survived was 11.07 millimeters long and 9.96 deep… The birds were not simply magnified by the drought: they were reformed and revised. They were changed by their dead. Their beaks were carved by their losses.” [p.78]

The drought was the crucible that shaped the species. And it wasn’t simply size, but dimension (longer and deeper beaks, versus wider) that separated the survivors from the dead.

In the same way, the benefits of new technologies are often masked during good times. Firms with both new and old technologies remain solidly profitable, happily hopping along. Like ornithologists watching finches in the wet season, some analysts have questioned whether technological innovation even matters. Robert Solow summed up this paradox by quipping “You can see the computer age everywhere but in the productivity statistics.”

But when hard times hit, innovators survive. More importantly, they flourish when the business cycle swings up again. Work by Erik Brynjolfsson and others has shown strong positive evidence for technology’s impact on productivity, most markedly over five-to-seven year periods – the resonant frequency of the business cycle. But like Darwin’s finches, the survivors are not just those who have more technology investments, but those who get the dimensions right.

Downturns are not only good for innovation, they are necessary.  While innovation may occur in times of plenty, crises allow the right innovations (hybrid cars) to outcompete the wrong ones (SUVs).  This assumes that crises are allowed to run their course (the case against bailouts), but that there are at least some survivors (the case for them).

As a data guy, I’m cautiously optimistic that firms who have invested in analytics, who have quietly innovated in understanding their business data, will emerge as winners on the other side of this downturn.  As a contemporary of Darwin’s said, “That which does not kill us makes us stronger.”

(April 2009 Update:  Unfortunately, The Money Tech Conference was indefinitely postponed, but fortunately I will be presenting a version of this talk in July at OSCON 2009).

I’ve been invited to speak at O’Reilly’s Money Tech conference this coming February 4-6th in New York City and thought I’d share the abstract for my talk here.  I’ll likely be in New York for several days, if you’d like to get together to chat about data drop me a line!

My talk is entitled “Open Source Analytics: Visualization and Predictive Modeling of Big Data with the R Programming Language”

ABSTRACT

Just as the explosion of online data catalyzed the development of
storage technologies such as Hadoop, new challenges in data analytics
– turning terabytes into actionable insights — demand new tools.  R,
an open-source language for statistical computing and graphics, is an
extensible, embeddable, and industry-strength solution for analytics.
In this session, I showcase R’s power by building predictive models
for Brazilian soybean harvests and baseball slugger salaries.

DESCRIPTION

The economics of data aggregation and analysis are being disrupted by
falling costs for storage and CPU power, the continuing shift of
business processes online, and the deluge of data that is being
generated as a consequence.

Satellite images, SEC filings, supply chain data (RFID data streams),
online prices, and newsgroup content represent just a few of the data
sources that hold potential for predictive modeling of markets.

Much of this data does not fit within existing paradigms for business
analysis: either its size overwhelms traditional desktop tools such as
Excel, or else its unique dimensions (such as geocodes) prevent its
being pipelined into more powerful, but narrowly designed, analysis
tools.  Finally, closed-source tools cannot keep pace with the leading
edge of innovation in statistical and machine-learning algorithms.

Enter the open source programming language R.  R has been dubbed the
lingua franca for statistical computing and graphical analysis, with a
pedigree tracing back several decades at Bell Labs.  Though its
million-plus users are concentrated within academia, R is gaining
currency within several high-profile quantitative analysis groups,
including Google’s Customer Insights team and Barclays Global
Investors.  In addition, R’s extensibility via user-contributed
packages has spawned an active developer community.

In this session, I will focus on applying R’s powerful visualization
tools to guide the construction of predictive models, using the kind
of large, multidimensional data sets that increasingly confront
quantitative analysts.  Along the way, I will highlight R’s packages
for inferential statistics, its compact modeling syntax, and its ease
of connectivity with persistent data stores.

The two specific examples I will discuss are:

- an analysis of NASA’s Landsat imagery of Brazil’s center-west
agricultural regions to detect correlates for soybean harvest yields,
and a derived predictor of the Brazilian soybean market based in part
on these correlates.

- a validation of Bill James’ sabermetrics approach to batting
performance using 30 years of Major League Baseball statistics, and a
derived predictor for batters’ salaries.

For all of its strengths, R has an admittedly steep learning curve.
While source code for the examples will be provided online, this talk
will emphasize techniques and working examples over technical details.
The goal of this session is to give quantitative analysts the courage
to invest in learning the R language, by showcasing R’s power,
highlighting its features, and providing examples of its use for
innovative applications.