How well do you know your friends’ political views? According to recent work by Winter Mason, Duncan Watts, and myself, you probably don’t know them as well as you think. In particular, we found that when friends disagree on a political issue, they are unaware of that disagreement about 60% of the time. Even close friends who discuss politics are typically unaware of their differences in opinions.

Homophily, the tendency for individuals to associate with similar others, is one of the more consistently observed phenomena of the social world. While reports of homophily have focused on socio-demographic attributes—like age, gender, and race—it’s reasonable that homophily would extend to attitudes as well, in part because individuals likely seek out those who agree with them, and in part because social pressure often breeds conformity. However, measuring the extent to which friends agree with one another is a tough empirical problem, and quantifying the differences between actual and perceived homophily was, until recently, prohibitively difficult.

Enter Facebook, the social science experimentation platform with a subject pool of 400 million strong (and growing). To examine the differences between real and perceived attitude agreement, we built a Facebook game where players both answered a series of questions about their own attitudes, and attempted to guess how particular friends would respond. We collected 12,000+ responses in which we had a player’s answer to a given question, their friend’s answer to that same question, and the player’s guess as to what their friend would say.

The figure below shows that people consistently overestimate the likelihood that their friends agree with them on political issues. Notably, even though close friends (so-called strong ties^[1]) are in reality more likely to agree with one another than distant friends, people do not appropriately adjust their perceptions. In other words, though we think close and distant friends are about equally likely to agree with us on political issues, in reality we are much more likely to agree with close friends.

Upon reflection, it’s perhaps not so surprising that we often don’t know where our friends stand on hot-button political issues. How many times have you actually asked your friends how they feel about abortion, capital punishment, or affirmative action? Inferring friends’ attitudes is further complicated by the fact that political positions tend to be only weakly correlated across issues. For example, according to the 2008 General Social Survey, 65% of people who support abortion rights also support capital punishment, compared to 68% among those who oppose abortions. That is, someone’s position on abortion tells you nearly nothing about their view on capital punishment.

I think society would benefit from friends and neighbors engaging in more active political deliberation, discussions that at the very least leave participants aware of each other’s views. On the other hand, if your friends really knew what you thought, you might have a lot fewer friends.

N.B. For more details, check out our paper.

Illustration by Kelly Savage

Footnotes

[1] We define the tie strength between a pair of individuals by considering the number of mutual friends that they have, and whether or not the pair discusses politics.

The idea of rot13 is to obscure text, for example, to prevent spoilers. It’s not meant to be cryptographically secure but simply to make sure that only people who are sure they want to read something will read it.

We, being pretty extreme nerds, sometimes find ourselves wanting to do something similar for numbers. This is typically in the context of a sealed-bid auction where you want to submit a bid that the other party can unseal whenever they’re ready, with no further coordination needed. Most recently, Sharad and I had to decide who would be presenting our Prediction Without Markets paper at the upcoming ACM Ecommerce conference. To determine who wanted to present more (i.e., who minded least), I sent him my sealed bid (how much I would pay him, at most, to present) and told him to unseal it when he had committed to his own bid. (Eliminating the need for further coordination was especially useful here because Sharad was about to get on a plane. It’s similarly handy when coordinating by text message when one or both parties are about to get on the subway.)

The outcome in that case was that Sharad bid lower and so I paid him that lower bid to be the one to present our paper.

In general what we want is a way for me to send you a number and trust you to pick your own number, uninfluenced by mine. And then you should be able to reveal mine when you’re ready, without requiring further input from me or any third party. Note the assumption that the recipient is being trusted not to cheat.

So how do we do that? It’s not as simple as rot13 because certain numbers, like 1 and 2, will recur often enough that you might remember that, say, “gjb” is really 2.

To be more specific, we seek a function, seal(), that maps a real number to a real number (or a string). It should not be deterministic — seal(7) should not map to the same thing every time. But the corresponding function, reveal(), should be deterministic — reveal(seal(x)) should equal x for all x.

For example:

   > seal(7)              # Some random-seeming
   429964                 # number or string.

   > seal(7)              # A completely different
   749932                 # random-seeming number.

   > reveal(seal(7))      # Revealing always uncovers
   7                      # the original number.

Can you think of a way to implement seal() and reveal()?

How to Seal and Reveal a Secret Number

There are many ways, it turns out, to accomplish this but here, in lay terms, is an elegant scheme that can be carried out easily with a pocket calculator:

Pick a random number (from your head is fine) between 9 and 99 and multiply it by 9999, then add your secret number. This will yield a 5 or 6-digit number that encodes your secret number. To decode it, divide by 9999, subtract the part to the left of the decimal point, then multiply by 9999. (This is known to children and mathematicians as “finding the remainder when dividing by 9999” and “mod’ing by 9999”, respectively.)

This works for nonnegative numbers less than 9999 (if that’s not enough, use 99999 or as many nines as you want). If you want to allow negative numbers, then the magic 9999 number needs to be twice the biggest possible number. And when decoding, if the result is greater than half of 9999, i.e., 5000 or more, then subtract 9999 to get the actual (negative) number.

Again, this is not a true commitment scheme. But for cases where the honor system suffices and the only goal is to not be influenced by each other’s number, we find this quite handy.

Pseudocode

   M = 9999  # Numbers between -M/2 and M/2 can be sealed.

   seal(x): M * randInt(9, 99) + x

   reveal(x): m = mod(x, M);
              if m > M/2 return m - M else return m

Image by Jeff Moser.

Thanks to Svante v. Erichsen for the idea behind this scheme, and to a dozen other Stack Overflow users for alternative schemes.

How do you estimate the prevalence of HIV among drug addicts? In a perfect (though somewhat Orwellian) world you’d have a giant list of all drug users, and you’d randomly select a subset of them to test for HIV. Alas, the world is messy, and we’re left to less rigorous methods, for example time-location sampling, where surveyors recruit drug users from the local “shooting gallery” (heroin users’ version of a crack house) at randomly selected times.

A few years ago, sociologists Doug Heckathorn and Matthew Salganik proposed a new approach to surveying these so-called hard-to-reach or hidden populations: respondent-driven sampling. RDS is a variant of snowball sampling—where existing study participants recruit the next wave of participants, usually in exchange for payment—with a few clever tweaks. Matt and Doug realized that popular drug users (i.e., those with a lot of friends) are disproportionately likely to be in the final snowball sample, for the simple reason that popular users know more people who could potentially recruit them. In RDS, recruits are thus down-weighted by the number of people they know. In just a few short years, RDS has already been applied in more than 20 countries, and it is currently used by the Centers for Disease Control and Prevention (CDC) to help track the HIV epidemic in the United States. Now for the bad news: Matt and I just wrote a paper showing that RDS may not be suitable for key aspects of public health surveillance where it is now extensively applied.

Our critique of RDS boils down to a simple empirical reality: outcomes are typically not typical. To give an example, while the mean household income in the United States is around $60K, half of households earn either less than $30K or more than $100K. In other words, households typically have incomes that are far from typical. So what does this have to do with RDS? The trick of down-weighting popular drug users guarantees—albeit under strong assumptions—that RDS will on average yield the true infection level. (In statistical parlance, RDS is an asymptotically unbiased estimator.) In the RDS community, this theoretical property was widely interpreted as meaning that RDS is a generally reliable method of estimation. What we show is that even in situations where RDS estimates are correct on average, these estimates are typically so far from the truth that meaningful statistical inference is difficult. To clarify the point, suppose we estimate the mean age in New York by sampling one person uniformly at random. Even though our estimate would be perfect on average (i.e., unbiased), it could hardly be called accurate. Since RDS participants tend to know, and hence recruit, people who are similar to themselves, this caricatured example is actually not so far from the truth about why RDS estimates are so volatile.

I suspect that this tendency to neglect the variance also plays a role in discriminatory attitudes. Consider the following statistic, culled from the 2008 General Social Survey: white Americans on average watch 2.8 hours of television daily, compared to 4.4 hours watched on average by black Americans. The difference in group means is large, both on a relative scale—blacks on average watch nearly 60% more television than whites—and on an absolute scale—1.6 hours a day. (Citing a similar statistic, conservative radio host, and Rush Limbaugh protege, Michael Medved has pontificated on the subject.) Now, if you’re walking down the street and you see a random black person and a random white person, how likely is it that the black person watches more TV? Given the large group differences, you might be tempted to conclude it’s pretty likely. As it turns out, however, individual variation is so large that it’s basically a coin flip (about 60%) as to who watches more TV—in other words, race is a poor cue. The plot below makes this phenomenon visually apparent: despite the substantial difference in group means, whites and blacks have distributions of TV watching that are tough to distinguish. Just as in RDS, the means are misleading.

NB: For more details on RDS, check out our paper.

Illustration by Kelly Savage: A stylized depiction of an RDS recruitment tree, where nodes correspond to sample members and links indicate who recruited whom. Based on an RDS study of drug users in New York City, presented in Abdul-Quader, et al, 2006.

218 of you took our calibration quiz, not counting the 10% of submissions that had to be thrown out for not being complete or giving ranges with the min greater than the max or other sanity check failures. (Here’s the raw data.)

The bad news is that you’re terrible at making 90% confidence intervals. For example, not a single person had all 10 of their intervals contain the true answer, which, if everyone were perfectly calibrated, should’ve happened by chance to 35% of you. Getting less than 6 good intervals should, statistically, not have happened to anyone. How many actually had 5 or fewer good intervals? 76% of you.

Here’s a histogram of the number of good intervals you got, out of 10:

The overlaid phantom histogram is what it would look like if it were really the case that every interval people gave had a 90% chance of containing the true answer. In other words, you should’ve made your intervals much wider. When we ask for a 90% confidence interval there’s in fact only a 41% chance that your interval contains the true answer.

We ran this quiz on Mechanical Turk as well and you marginally outperformed the turkers. The histogram of turkers’ good intervals is indicated by the red dots in the above graph. They failed our sanity checks at almost twice the rate (19%) of Messy Matters readers and of the remaining responses, the mean number of good intervals was 3.5 out of 10.

The more we’ve thought about (and read the literature on — or rather, consulted endlessly with Dan Goldstein, who knows the literature on) these kinds of overconfidence results, however, the less clear it is that the moral of this quiz is simply “people are overconfident”. For one thing, overconfidence depends on the question. The fraction of good intervals in your responses ranged from 23% (the length of the Nile and the gestation period of an Asian elephant) to 75% (number of OPEC countries). Of course, even 75% is not the 90% that was asked for.

More interestingly, in an ongoing follow-up study on Mechanical Turk we’re finding that after you get people’s intervals, more than half of them realize in retrospect that too few of their intervals are good. This suggests that people can learn to perform much better at this task.

Obligatory Wisdom of Crowds Demonstration

It’s not a fair demonstration since people weren’t asked for their best guesses, but here’s a table of median lower bounds, upper bounds, and midpoints of everyone’s ranges. Interestingly, people’s upper bounds are overall most accurate.

Thanks to Sharad Goel, Dan Goldstein, Bethany Soule, Dan Kaminsky, and Michael J.J. Tiffany.

Image: Kelly Savage

We shall now find out if Messy Matters readers are smarter than Mechanical Turkers. For each of the questions below, provide a numerical range that you are 90% sure contains the correct answer. In particular, if you have “no idea” then give a very wide range; and if you happen to be quite certain then give a narrow range.

If you are perfectly calibrated then 90% of your intervals (no more, no less) should contain the right answers.

Please try your best, but don’t look up the answers!

(If the quiz doesn’t show up in your news reader, click through to the post on messymatters.com.)

We’ll discuss the results in the next post.

PS: Here are the results.

Quiz from Decision Traps by Russo and Shoemaker.
Image by Kelly Savage.

In the 2008 Summer Olympics Usain Bolt ran 100 meters in 9.69 seconds, earning the gold medal and garnering the international attention that comes with being the “fastest man in the world.” While Bolt became a household name, his competitors didn’t fare nearly as well: far fewer people know that Richard Thompson and Walter Dix received silver and bronze, and I suspect that 8th place Darvis Patton is practically unknown outside the sprinting world. The 340 milliseconds that separated Bolt from Patton—the duration of a blink of the eye—was the difference between celebrity and obscurity. While a fascination with rank is perhaps justifiable for sports, such focus—let’s call it the Gold Medal Syndrome—is often problematic in statistical analysis.

Consider the case of prediction markets. In these markets, participants buy and sell securities that realize a value based on the occurrence of some future outcome, such as the result of an election, the box office revenue of an upcoming film, or the market share of a new product. For example, the day before the 2008 U.S. presidential election you could have paid $0.92 for a contract in the Iowa Electronic Markets that yielded $1 when Barack Obama won, implying a 92% market-estimated probability that Obama would win. There are compelling theoretical reasons to expect prediction markets to outperform all other available forecasting methods. As formalized by the efficient-market hypothesis, if there were some way to beat the market then at least one savvy trader would presumably exploit that advantage to make money; hence, market prices should update to eliminate any performance disparity.

Inspired by such theoretical arguments, and also by a growing body of empirical findings that show markets beat alternatives, several researchers have called for widespread application of prediction markets to real-world business strategy and policy development problems. In a 2007 Wall Street Journal op-ed, economists Robert Hahn and Paul Tetlock write:

“Imagine the president had a crystal ball to predict more accurately the impact of broader prescription coverage on the Medicare budget, the effect of more frequent audits on tax compliance—or even the consequences of a political settlement in Iraq on oil prices. Now, stop imagining: Such crystal balls [prediction markets] are within our grasp.”

The work on which these appeals are based, however, primarily addresses the relative ranking of prediction methods. By contrast, the magnitude of the differences in question has received much less attention, and as such, it remains unclear whether the performance improvement associated with prediction markets is meaningful from a practical perspective.

In a new study, Daniel Reeves, Duncan Watts, Dave Pennock and I compare the performance of prediction markets to conventional means of forecasting, namely polls and statistical models. Examining thousands of sporting and movie events, we find that the relative advantage of prediction markets is remarkably small. For example, the Las Vegas market for professional football is only 3% more accurate in predicting final game scores than a simple, three parameter statistical model, and the market is only 1% better than a poll of football enthusiasts. The plot below shows how the three methods perform on the complementary task of estimating the probability the home team wins.

Given that sports and entertainment markets are among the most mature and successful, our results challenge the view that prediction markets are substantively superior to alternative forecasting mechanisms. Nevertheless, it is certainly possible that there are forecasting applications where either the relative advantage of markets is larger, or that such differences in performance are consequential. Thus, while prediction markets may yet prove to be useful, it would seem the enthusiasm for their predictive prowess has outpaced the evidence.

NB: Check out our paper for more details.

Illustration by Kelly Savage

As a poser economist (as Jeff Ely calls me), I love to complain about the social inefficiency of gift giving. It’s a terrible idea, guys! We waste 13 billion dollars a year on it! ^[1] But I do appreciate that I have a tendency to pooh-pooh social conventions far too blithely. Nonetheless, social conventions change. Drastically so, given enough time. As Al Roth points out, wedding registries, gift receipts, and, more regrettably, gift cards represent a shift towards cash as gifts, and thus greater efficiency.

But as much as I applaud this trend and agree that holiday gift giving makes us all poorer and generally worse off, I’m sure Joel Waldfogel’s estimate (that the value we get from gifts is 25 cents on the dollar) is overblown. So, in the spirit of the holiday season, here are some reasons why gift giving isn’t quite so bad as it seems:

1. Ulterior Motives. When put that way it doesn’t exactly sound holiday spirited but I think ulterior motives is the most overlooked and perhaps most legitimate justification for gift giving. If I give you a ticket to see a show with me because I really want to see it with you, then it really doesn’t matter how much you would’ve paid for that ticket yourself. I wanted your company and this is a socially acceptable way for me to pay for it. ^[2] Homer Simpson giving Marge a bowling ball (or a power screwdriver or somesuch) is another example. Nothing’s wasted if Homer gets enough use out of those gifts himself! A more common example is giving one’s girlfriend lingerie. But the ulterior motives need not be selfish, exactly. It might well be socially efficient for me to buy you a book you wouldn’t have bought, if I have a lot of value for discussing it with you.
2. Signaling Value. I don’t pretend to understand the intricacies of this, but presumably it’s socially useful to prove our thoughtfulness. We may also find it socially valuable to be mutually indebted to each other (and in ways hard to quantify).
3. The Ritual. The process itself may have value to us, including the process of figuring out each other’s desires well enough to buy gifts.
4. Extrospection. In rare cases you might actually know the recipient’s utility function better than they do. Usually this is ill-advised, from an efficiency standpoint. Why not just recommend your new favorite music to your friend and let them decide whether to buy it? The one case where extrospection is a slam dunk, of course, is giving gifts to small children (all the other excuses for gift giving — except perhaps number 2 — also apply in this case).
5. Transactional Efficiency. Related to number 4, it may be easier for you to get your friend something than it is for them to get it. Maybe you’re buying a book for yourself and it’s just as easy to get a second copy. Or you may have access to something the recipient doesn’t — for example, if you’re visiting a foreign country.

The last two don’t really count, of course, since Waldfogel’s analysis accounts for those. But the first three may put a legitimate dent in that 13 billion dollar deadweight loss.

Added: Vincent Conitzer adds a great one in the comments:

6. Plausible Deniability. There are some things that you just have embarrassingly high utility for. And “hey, it was a gift!” can be a nice airtight excuse.

Addendum: Other Inefficiencies of the Season

Sharad pointed out that “after-Christmas sale” is a euphemism for “pre-Christmas price-gouging.” This is not an inefficiency if you include the utility of the retailers but from consumers’ perspective, it’s inefficient to concentrate demand around a single day of the year. Bah humbug!

I recently encountered an inefficiency related to holiday travel. Many outbound NYC flights were canceled the weekend before Christmas due to a snowstorm. People were generally rebooked a few days later. But on Monday everything was back to normal. I know because I flew out of New York on Monday. My aunt was supposed to fly out on Saturday but got stranded until Tuesday. She clearly wanted to depart on Monday more than we did. It’s a shame that there’s essentially no secondary market for airline tickets! (Though there is in a very limited sense: You can volunteer to be bumped and the airline will compensate you with travel vouchers, worth deceptively little. But naturally the airlines are not bumping willing travelers in order to rebook people sooner when their original flights were canceled.)

Footnotes

[1] That’s not a claim that we spend $13B/year on gifts. We spend 17 billion, 13 billion of which is pure waste. That’s Joel Waldfogel’s estimate of the difference between what we spend on holiday gifts and what the recipients would have been willing to spend to buy those items themselves.

[2] Alright, it sounds much ickier when you put it that way!

Thanks to Alex Strehl, Eleanor Strehl, David Reiley, Jill Sazama, Bethany Soule, and Sharad Goel for helpful discussion.

Illustration by Kelly Savage

Statistical wisdom is sometimes found in unusual places. Take, for example, the following exchange in, “Whip It,” a story of roller-derby loving misfits:

Team: (chanting, after a game) We’re number two! We’re number two!
Coach: You guys came in second out of two teams.
Team: Woo!

So what’s the moral? Well, there’s been a fair amount of buzz this year over the finding that web search behavior is correlated with offline outcomes, including flu incidence and economic activity (e.g., auto and home sales). For example, an increase in queries for “flu” and “cold” is often associated with a rise in actual flu caseloads. The correlation between search volume and flu levels in fact seems quite good, hovering around 0.95. My initial reaction to these results was somewhat critical, and boils down to the observation that simple statistical models are comparable to, often even better than, search-based predictions. Invoking the first lesson of Whip It: Success is best evaluated in context. To give another example of this principle, it may sound impressive to predict with 97% accuracy whether it’ll be cloudy in Cold Bay, AK  — at least until you learn Cold Bay is only sunny 10 days a year. This assessment, however, should be weighed against the second lesson of Whip It: Number two can still be cool. The fact that search is so highly correlated with health and economic outcomes says something interesting about human behavior, regardless of whether or not it’ll help save the world.

Motivated by these examples, fellow yahoos (Jake Hofman, Sébastien Lahaie, Dave Pennock, and Duncan Watts) and I recently investigated the extent to which search behavior predicts the success of cultural products, namely movies, video games, and music. In a departure from past work that has focused on real-time reporting of current activity (e.g., flu trends) — what Choi and Varian cleverly call “predicting the present” — our objective was to predict future events, typically days to weeks in advance. Specifically, we use query volume to forecast opening weekend box-office revenue for feature films, first month sales of video games, and the rank of songs on the Billboard Hot 100. In all cases that we consider, we find search volume on its own is predictive of future outcomes, but search is nevertheless outperformed by baseline models trained on publicly available data; combining search and baseline models generally leads to modest improvements.

Whether web search is useful in predicting real-world activity is therefore likely a matter of circumstance and necessity: On the one hand, across a variety of domains, adding search to baseline models does not dramatically boost performance; but on the other hand, in certain situations, such baseline estimates may be difficult to generate, and for some applications, even small gains in performance are valuable. In other words, the benefit of web search as a prediction tool may have less to do with its superiority over other methods than with its generality, low cost, and real-time nature.

NB: For more details, check out our paper.

Illustration by Kelly Savage

Grammatical prescriptivism annoys me to no end. (No ending sentences with prepositions? Ridiculous! Don’t split infinitives? Poppycock!) Well, except when I do it…

There are exactly two valid uses of quotation marks: when you mean something literally, and when you mean something nonliterally. It ought to be hard to mess that up.  But the following are not valid uses:

0. I really, truly, extra mean this!
   I wasn’t going to mention this one because I presume that you (dear reader of Messy Matters) have never done this, but the blog of unnecessary quotes is just too funny. And I suppose it is a bit confusing: italics can sometimes serve the function of quotation marks (read on) but not the other way around.

On to more the more literate misuses (admit it, you’ve done these):

1. I’m not going to tell you what I mean by this.
   E.g., “the result is ‘nearly’ optimal” by which the author means “the result is nearly optimal, for some definition of ‘nearly’, which I can’t be bothered to specify.”
2. This sounds too colloquial/informal/idiomatic.
   E.g., “The subjects got ‘carried away’.” The author here is thinking, “this sounds too colloquial or idiomatic for formal writing but it’s OK because I’m not really writing it, I’m just quoting a more informal version of myself.” Please. Just write it or don’t write it. The only exception is when you expect the idiom to be unknown to the reader. In that case, the second acceptable usage below applies.
3. I’m coining this term/phrase or introducing terminology.
   Use italics instead. (“We say that an allocation is satisficing if…”) Exception: it is acceptable if the usage is also literal. (“We use the term ‘satisficing’ to refer to…” Though italics are acceptable here as well.)

Which brings us to the two acceptable usages:

1. I mean this literally. I’m referring to the word or phrase itself.
   E.g., “‘Promulgate’ is hard to pronounce.” Or, “Alice said ‘Look out!’” Another instance of this rule is referring to works by title, with the slew of exceptions that certain titles, like books and movies, should be italicized rather than quoted. There are also other times when italics may substitute for quotation marks, such as in the first example with the word promulgate. ^[1]
2. I don’t mean this literally.
   You can use scare quotes when something is not the right word or phrase. Of course what I really mean by “not the right word” is that it simultaneously is and isn’t the right word. E.g., “the player then ‘wins’ with a loss of $100” (they technically won but really they lost). Or if the word is accurate only loosely by analogy, e.g., “the program bids ‘spitefully’” (there’s a tit-for-tat aspect to its bidding but, being a computer, it doesn’t actually have spite). Wrapping something in scare quotes is like adding a parenthetical “not really”. It should be done sparingly; typically only when the nonliteral usage might be confusing if not “called out”.

P.S. If the quotation marks in the last sentence were like fingernails on a blackboard to you then you didn’t need to read any of this. And, ironically, you’re likely the only one who didn’t find this all a painfully pedantic waste of time!

EDIT: Per Hartman’s Law of Prescriptivist Retaliation I had to fix a couple errors since this went to press (embarrassingly, “slough” instead of “slew”). Also, thank you to Eva Revesz, Kevin Lochner, and Reema Bennekaa for commenting on drafts of this.

Illustration by Kelly Savage.

Footnote:

[1] There’s little obvious rhyme or reason to these quotes-vs-italics rules but I believe the principle is just this: Both quotation marks and italics are serving the same functional purpose (marking a word or phrase as literal) but with different forms to suggest various dichotomies, such as book titles vs. chapters within a book. Although even I couldn’t bring myself to care if you violated these conventions, as long as you’re internally consistent.