We shall now find out if Messy Matters readers are smarter than Mechanical Turkers. For each of the questions below, provide a numerical range that you are 90% sure contains the correct answer. In particular, if you have “no idea” then give a very wide range; and if you happen to be quite certain then give a narrow range.

If you are perfectly calibrated then 90% of your intervals (no more, no less) should contain the right answers.

Please try your best, but don’t look up the answers!

(If the quiz doesn’t show up in your news reader, click through to the post on messymatters.com.)

We’ll discuss the results in the next post.

Quiz from Decision Traps by Russo and Shoemaker.
Image by Kelly Savage.

In the 2008 Summer Olympics Usain Bolt ran 100 meters in 9.69 seconds, earning the gold medal and garnering the international attention that comes with being the “fastest man in the world.” While Bolt became a household name, his competitors didn’t fare nearly as well: far fewer people know that Richard Thompson and Walter Dix received silver and bronze, and I suspect that 8th place Darvis Patton is practically unknown outside the sprinting world. The 340 milliseconds that separated Bolt from Patton—the duration of a blink of the eye—was the difference between celebrity and obscurity. While a fascination with rank is perhaps justifiable for sports, such focus—let’s call it the Gold Medal Syndrome—is often problematic in statistical analysis.

Consider the case of prediction markets. In these markets, participants buy and sell securities that realize a value based on the occurrence of some future outcome, such as the result of an election, the box office revenue of an upcoming film, or the market share of a new product. For example, the day before the 2008 U.S. presidential election you could have paid $0.92 for a contract in the Iowa Electronic Markets that yielded $1 when Barack Obama won, implying a 92% market-estimated probability that Obama would win. There are compelling theoretical reasons to expect prediction markets to outperform all other available forecasting methods. As formalized by the efficient-market hypothesis, if there were some way to beat the market then at least one savvy trader would presumably exploit that advantage to make money; hence, market prices should update to eliminate any performance disparity.

Inspired by such theoretical arguments, and also by a growing body of empirical findings that show markets beat alternatives, several researchers have called for widespread application of prediction markets to real-world business strategy and policy development problems. In a 2007 Wall Street Journal op-ed, economists Robert Hahn and Paul Tetlock write:

“Imagine the president had a crystal ball to predict more accurately the impact of broader prescription coverage on the Medicare budget, the effect of more frequent audits on tax compliance—or even the consequences of a political settlement in Iraq on oil prices. Now, stop imagining: Such crystal balls [prediction markets] are within our grasp.”

The work on which these appeals are based, however, primarily addresses the relative ranking of prediction methods. By contrast, the magnitude of the differences in question has received much less attention, and as such, it remains unclear whether the performance improvement associated with prediction markets is meaningful from a practical perspective.

In a new study, Daniel Reeves, Duncan Watts, Dave Pennock and I compare the performance of prediction markets to conventional means of forecasting, namely polls and statistical models. Examining thousands of sporting and movie events, we find that the relative advantage of prediction markets is remarkably small. For example, the Las Vegas market for professional football is only 3% more accurate in predicting final game scores than a simple, three parameter statistical model, and the market is only 1% better than a poll of football enthusiasts. The plot below shows how the three methods perform on the complementary task of estimating the probability the home team wins.

Given that sports and entertainment markets are among the most mature and successful, our results challenge the view that prediction markets are substantively superior to alternative forecasting mechanisms. Nevertheless, it is certainly possible that there are forecasting applications where either the relative advantage of markets is larger, or that such differences in performance are consequential. Thus, while prediction markets may yet prove to be useful, it would seem the enthusiasm for their predictive prowess has outpaced the evidence.

NB: Check out our paper for more details.

Illustration by Kelly Savage

As a poser economist (as Jeff Ely calls me), I love to complain about the social inefficiency of gift giving. It’s a terrible idea, guys! We waste 13 billion dollars a year on it! ^[1] But I do appreciate that I have a tendency to pooh-pooh social conventions far too blithely. Nonetheless, social conventions change. Drastically so, given enough time. As Al Roth points out, wedding registries, gift receipts, and, more regrettably, gift cards represent a shift towards cash as gifts, and thus greater efficiency.

But as much as I applaud this trend and agree that holiday gift giving makes us all poorer and generally worse off, I’m sure Joel Waldfogel’s estimate (that the value we get from gifts is 25 cents on the dollar) is overblown. So, in the spirit of the holiday season, here are some reasons why gift giving isn’t quite so bad as it seems:

1. Ulterior Motives. When put that way it doesn’t exactly sound holiday spirited but I think ulterior motives is the most overlooked and perhaps most legitimate justification for gift giving. If I give you a ticket to see a show with me because I really want to see it with you, then it really doesn’t matter how much you would’ve paid for that ticket yourself. I wanted your company and this is a socially acceptable way for me to pay for it. ^[2] Homer Simpson giving Marge a bowling ball (or a power screwdriver or somesuch) is another example. Nothing’s wasted if Homer gets enough use out of those gifts himself! A more common example is giving one’s girlfriend lingerie. But the ulterior motives need not be selfish, exactly. It might well be socially efficient for me to buy you a book you wouldn’t have bought, if I have a lot of value for discussing it with you.
2. Signaling Value. I don’t pretend to understand the intricacies of this, but presumably it’s socially useful to prove our thoughtfulness. We may also find it socially valuable to be mutually indebted to each other (and in ways hard to quantify).
3. The Ritual. The process itself may have value to us, including the process of figuring out each other’s desires well enough to buy gifts.
4. Extrospection. In rare cases you might actually know the recipient’s utility function better than they do. Usually this is ill-advised, from an efficiency standpoint. Why not just recommend your new favorite music to your friend and let them decide whether to buy it? The one case where extrospection is a slam dunk, of course, is giving gifts to small children (all the other excuses for gift giving — except perhaps number 2 — also apply in this case).
5. Transactional Efficiency. Related to number 4, it may be easier for you to get your friend something than it is for them to get it. Maybe you’re buying a book for yourself and it’s just as easy to get a second copy. Or you may have access to something the recipient doesn’t — for example, if you’re visiting a foreign country.

The last two don’t really count, of course, since Waldfogel’s analysis accounts for those. But the first three may put a legitimate dent in that 13 billion dollar deadweight loss.

Added: Vincent Conitzer adds a great one in the comments:

6. Plausible Deniability. There are some things that you just have embarrassingly high utility for. And “hey, it was a gift!” can be a nice airtight excuse.

Addendum: Other Inefficiencies of the Season

Sharad pointed out that “after-Christmas sale” is a euphemism for “pre-Christmas price-gouging.” This is not an inefficiency if you include the utility of the retailers but from consumers’ perspective, it’s inefficient to concentrate demand around a single day of the year. Bah humbug!

I recently encountered an inefficiency related to holiday travel. Many outbound NYC flights were canceled the weekend before Christmas due to a snowstorm. People were generally rebooked a few days later. But on Monday everything was back to normal. I know because I flew out of New York on Monday. My aunt was supposed to fly out on Saturday but got stranded until Tuesday. She clearly wanted to depart on Monday more than we did. It’s a shame that there’s essentially no secondary market for airline tickets! (Though there is in a very limited sense: You can volunteer to be bumped and the airline will compensate you with travel vouchers, worth deceptively little. But naturally the airlines are not bumping willing travelers in order to rebook people sooner when their original flights were canceled.)

Footnotes

[1] That’s not a claim that we spend $13B/year on gifts. We spend 17 billion, 13 billion of which is pure waste. That’s Joel Waldfogel’s estimate of the difference between what we spend on holiday gifts and what the recipients would have been willing to spend to buy those items themselves.

[2] Alright, it sounds much ickier when you put it that way!

Thanks to Alex Strehl, Eleanor Strehl, David Reiley, Jill Sazama, Bethany Soule, and Sharad Goel for helpful discussion.

Illustration by Kelly Savage

Statistical wisdom is sometimes found in unusual places. Take, for example, the following exchange in, “Whip It,” a story of roller-derby loving misfits:

Team: (chanting, after a game) We’re number two! We’re number two!
Coach: You guys came in second out of two teams.
Team: Woo!

So what’s the moral? Well, there’s been a fair amount of buzz this year over the finding that web search behavior is correlated with offline outcomes, including flu incidence and economic activity (e.g., auto and home sales). For example, an increase in queries for “flu” and “cold” is often associated with a rise in actual flu caseloads. The correlation between search volume and flu levels in fact seems quite good, hovering around 0.95. My initial reaction to these results was somewhat critical, and boils down to the observation that simple statistical models are comparable to, often even better than, search-based predictions. Invoking the first lesson of Whip It: Success is best evaluated in context. To give another example of this principle, it may sound impressive to predict with 97% accuracy whether it’ll be cloudy in Cold Bay, AK  — at least until you learn Cold Bay is only sunny 10 days a year. This assessment, however, should be weighed against the second lesson of Whip It: Number two can still be cool. The fact that search is so highly correlated with health and economic outcomes says something interesting about human behavior, regardless of whether or not it’ll help save the world.

Motivated by these examples, fellow yahoos (Jake Hofman, Sébastien Lahaie, Dave Pennock, and Duncan Watts) and I recently investigated the extent to which search behavior predicts the success of cultural products, namely movies, video games, and music. In a departure from past work that has focused on real-time reporting of current activity (e.g., flu trends) — what Choi and Varian cleverly call “predicting the present” — our objective was to predict future events, typically days to weeks in advance. Specifically, we use query volume to forecast opening weekend box-office revenue for feature films, first month sales of video games, and the rank of songs on the Billboard Hot 100. In all cases that we consider, we find search volume on its own is predictive of future outcomes, but search is nevertheless outperformed by baseline models trained on publicly available data; combining search and baseline models generally leads to modest improvements.

Whether web search is useful in predicting real-world activity is therefore likely a matter of circumstance and necessity: On the one hand, across a variety of domains, adding search to baseline models does not dramatically boost performance; but on the other hand, in certain situations, such baseline estimates may be difficult to generate, and for some applications, even small gains in performance are valuable. In other words, the benefit of web search as a prediction tool may have less to do with its superiority over other methods than with its generality, low cost, and real-time nature.

NB: For more details, check out our paper.

Illustration by Kelly Savage

Grammatical prescriptivism annoys me to no end. (No ending sentences with prepositions? Ridiculous! Don’t split infinitives? Poppycock!) Well, except when I do it…

There are exactly two valid uses of quotation marks: when you mean something literally, and when you mean something nonliterally. It ought to be hard to mess that up.  But the following are not valid uses:

0. I really, truly, extra mean this!
   I wasn’t going to mention this one because I presume that you (dear reader of Messy Matters) have never done this, but the blog of unnecessary quotes is just too funny. And I suppose it is a bit confusing: italics can sometimes serve the function of quotation marks (read on) but not the other way around.

On to more the more literate misuses (admit it, you’ve done these):

1. I’m not going to tell you what I mean by this.
   E.g., “the result is ‘nearly’ optimal” by which the author means “the result is nearly optimal, for some definition of ‘nearly’, which I can’t be bothered to specify.”
2. This sounds too colloquial/informal/idiomatic.
   E.g., “The subjects got ‘carried away’.” The author here is thinking, “this sounds too colloquial or idiomatic for formal writing but it’s OK because I’m not really writing it, I’m just quoting a more informal version of myself.” Please. Just write it or don’t write it. The only exception is when you expect the word or phrase to be unknown to the reader. In that case, the second acceptable usage below applies.
3. I’m coining this term/phrase or introducing terminology.
   Use italics instead. (“We say that an allocation is satisficing if…”) Exception: it is acceptable if the usage is also literal. (“We use the term ‘satisficing’ to refer to…” Though italics are acceptable here as well.)

Which brings us to the two acceptable usages:

1. I mean this literally. I’m referring to the word or phrase itself.
   E.g., “‘Promulgate’ is hard to pronounce.” Or, “Alice said ‘Look out!’” Another instance of this rule is referring to works by title, with the slew of exceptions that certain titles, like books and movies, should be italicized rather than quoted. There are also other times when italics may substitute for quotation marks, such as in the first example with the word promulgate. ^[1]
2. I don’t mean this literally.
   You can use scare quotes when something is not the right word or phrase. Of course what I really mean by “not the right word” is that it simultaneously is and isn’t the right word. E.g., “the player then ‘wins’ with a loss of $100” (they technically won but really they lost). Or if the word is accurate only loosely by analogy, e.g., “the program bids ‘spitefully’” (there’s a tit-for-tat aspect to its bidding but, being a computer, it doesn’t actually have spite). Wrapping something in scare quotes is like adding a parenthetical “not really”. It should be done sparingly; typically only when the nonliteral usage might be confusing if not “called out”.

P.S. If the quotation marks in the last sentence were like fingernails on a blackboard to you then you didn’t need to read any of this. And, ironically, you’re likely the only one who didn’t find this all a painfully pedantic waste of time!

EDIT: Per Hartman’s Law of Prescriptivist Retaliation I had to fix a couple errors since this went to press (embarrassingly, “slough” instead of “slew”). Also, thank you to Eva Revesz, Kevin Lochner, and Reema Bennekaa for commenting on drafts of this.

Illustration by Kelly Savage.

Footnote:

[1] There’s little obvious rhyme or reason to these quotes-vs-italics rules but I believe the principle is just this: Both quotation marks and italics are serving the same functional purpose (marking a word or phrase as literal) but with different forms to suggest various dichotomies, such as book titles vs. chapters within a book. Although even I couldn’t bring myself to care if you violated these conventions, as long as you’re internally consistent.

I admit I’m a movie snob. I’m less likely to watch blockbusters about toys gone wild that open on 4000+ screens than tales of food gone wild that open on 50 screens. And for this I always figured I was among a minority who ventured beyond the standard Hollywood fare. Well, it turns out that it’s pretty ordinary to have extraordinary tastes. In a study of consumer preferences across movies, music, web browsing, and web search, Andrei Broder, Evgeniy Gabrilovich, Bo Pang, and I find that nearly everyone is a little bit eccentric.

The explosion of electronic commerce has opened the door to infinite-inventory retailers, such as Amazon and Netflix, that offer an order of magnitude more items than their brick-and-mortar counterparts.  These so-called “Long Tail” markets, as coined by Chris Anderson, have been found to exhibit two near-universal properties: First, the vast majority of products are “misses,” appealing to only a relatively small group of people; and second, these “worst-sellers” in aggregate account for a sizable fraction of total consumption. For example, 30% of Amazon’s sales and 25% of Netflix’s sales are for items not available in the largest offline retail stores. Based on these empirical observations, the success of online retailers has been largely attributed to the lucrative, and previously untapped, market for niche items.

The long tail phenomenon, though, is in principle consistent with two fundamentally different hypotheses. The conventional wisdom is that most people exclusively consume popular offerings, with only a minority seeking out niche content. However, a second hypothesis is that everyone is a little eccentric, consuming both popular and specialty products. Importantly, these two theories predict substantively different consequences of inventory size on consumer satisfaction. In the former case, a small inventory of high-demand items would satisfy most people nearly all of the time, while in the latter, such an inventory would frustrate most people at least some of the time.

We find overwhelming evidence that nearly everyone is at least a bit eccentric. For example, 85% of Netflix users and 95% of Yahoo! Music users have ventured into the tail (i.e., consumed items not available in large, brick-and-mortar retailers), and 40% of Netflix users and 70% of Yahoo! Music users regularly consume tail items. Our results thus suggest an additional factor for the success of infinite-inventory retailers. Tail inventory may boost head sales by offering consumers the convenience of “one-stop shopping” for both their mainstream and niche interests. Hence, even small increases in direct revenue from niche products may be associated with much larger second-order gains due to increased overall consumer satisfaction and resulting repeat patronage.

One can see this second-order effect by examining the relationship, as inventory size increases, between the percent of demand fulfilled and the percent of consumers satisfied. Stocking more items naturally leads to selling more items (i.e., the first-order effect of directly fulfilling demand). What the plot above indicates, however, is that increases in direct revenue come with disproportionate increases in consumer satisfaction. Consequently, to the extent that increased inventory attracts new customers, direct revenue calculations undervalue the tail. In particular, when deciding whether or not to stock the latest worst-seller, the pertinent question is not, How many copies will I sell?, but rather How many customers will I gain?.

So people aren’t sheep after all. That’s probably a good thing, but it makes it a whole lot harder to run a video store.

N.B. For more details, check out our paper. Anita Elberse also discusses related findings in her Harvard Business Review article, Should You Invest in the Long Tail?.

Illustration: Kelly Savage