So we make lists. Lists are an attempt to curate Twitter experiences by categorizing people according to whatever topic you choose, so that viewing tweets from that list will give you a more focused view of the tweetstream. I have noticed that I am popping up on lists about media. Sometimes I tweet about media, so that makes sense, and I am grateful to be included on those lists.

But people are messy, and don’t go back into boxes very well. For every tweet about media I might have three about market research, six funny videos, 25 out-of-context replies to others and 50 pictures of my son’s upcoming 5th birthday party (Happy Birthday, S!) This is likely to be true of nearly everyone else on these lists, unless you just use Twitter as a blast mechanism for your business (and Twitter won’t grow that way!)

So I might find myself on lists for market research, branding, radio, social media, etc. At any given time, chances are whatever list you see me on has little to do with my last tweet, and maybe my last 20 tweets. Still, let’s assume that Twitter lists will impose a little order. Lists are a new toy, and people are starting to follow them en masse. When you follow a few lists, you are potentially following a lot more people than you used to. Given my thesis that most humans will not fit neatly into those lists, it may be that adding 4 or 5 “focused” lists (especially ones that you didn’t make yourself) to your follower counts will make Twitter more chaotic, not less.

I prefer saved searches to lists any day. What do you think?

I love focus groups. There are detractors out there, but I will only point out that I have been able to give clients insights into why their customers and prospects behave the way they do that would simply not be possible to ascertain via other means, including social media monitoring. At their best, focus groups are part of a coordinated plan that also includes quantitative validation and a mix of monitoring/listening techniques, because no one methodology can do it all.

The problem I have faced recently is “feature creep”: when other stakeholders in an organization sniff out the fact that focus groups are being done, everyone sees an opportunity to slip in some questions pertaining to their function or area of interest. Focus groups are awesome for generating ideas on a topic–for going through checklists of questions, not so much. It’s hard to fault anyone in an organization for this, especially one that has very few research inputs–when you have no other way to determine awareness for marketing initiative or promotion, it’s human nature to want to ask those sorts of questions in focus groups if you are doing them. But focus groups are lousy at the quantitative questions–if you send me in as a moderator with a checklist of questions like “did you get this in the mail” or “have you seen this TV spot” you are not only getting back “numbers” that you can’t trust, you are also suffering the opportunity cost of not using that time to pursue other types of exercises that focus groups really shine at.

This generally puts me in the role of triage nurse, and sometimes even bad cop. Often I will sit on a conference call with stakeholders to determine the direction of a study and be placed into the uncomfortable position of telling one or more stakeholders that this isn’t the right setting to find out what they need to know, or that we have to restrict the scope of the topic guide. That pains me–especially because I know that some of their questions might just go unanswered forever. It is always a struggle to balance the credo that the “customer is always right” with the fact that sometimes..they aren’t. My best clients get that–and appreciate when I set limits or boundaries on a project, because they know that the end results will be richer as a result. Sadly, with research budgets in many traditional media clients getting cut, I’m having to be bad cop more often, which is psychically taxing.

Sometimes I wonder if the bad rap that focus groups sometimes get in the branding community is due more to this–to the fact that their individual projects have been asked to do too much–than to any kind of methodological defect with focus groups themselves. I can’t quibble with people who have had bad experiences with focus groups; I’m sure these experiences are real. But, in some cases (especially in research-starved organizations,) it’s a poor potter who blames the clay.

This viewpoint is fairly common in the social media space at the moment–the evolving discovery that in a world of media multitasking and passive measurement for passive media, we may be able to measure how many drivers saw a billboard, but where’s the engagement? You can audit radio with passive measures, but who cares about the messages they hear?

I’ve also heard it said in the social media space that the only metric that really matters is what rings the till. That is the language, ultimately, that the C-level speaks, so all we need to do is link a social media initiative directly to increased sales, and the effort is justified. True enough, I suppose, but pursing some kind of direct linkage metric may prove to be a bit of a rathole.

The path to a sale is something like a romance. Regardless of whether or not you subscribe to the classic AIDA continuum or the Cognition-Affect-Behavior model, consumers generally digest multiple messages at multiple touchpoints in the process of formulating their own want, need or desire for a product. Hearing about a product a few times on the radio might spark the initial “What?”, in-store marketing/trial might provide some kind of factual proof, online reviews social proof and so on. However you examine the process, most people become dimly aware of products, then dimly aware of how a product might fit into their lives (emotionally or logically) and then move into deeper levels of engagement with a product or service before they decide to buy. Social media undoubtedly belongs in that picture. In fact, social media might just be the most powerful variable in terms of changing a consumer’s willingness to consider a brand (there’s a metric!) or perception of brand fit (there’s another!). But if social media moves a potential buyer from apathy to engagement with a brand, but a Val-Pak coupon is the last step in the “sale,” social media will continue to sit at the kid’s table as far as “ringing the till” is concerned.

Social media initiatives cannot be allowed to devolve into direct sales efforts. The early adopters are right to point out that traditional media’s “audit” style metrics are falling off a cliff. But there are concerted efforts underway now to develop models for engagement and new metrics for all manner of non-traditional forms of marketing and advertising. Smart marketers use any and all tools at their disposal to engage consumers at multiple points along the consideration continuum. Do urinal ads lead to direct sales? I don’t know where you pee, but probably not. Are they measured? You betcha. A display in the mens’ room for a product changes perception of that brand, gives it personality and potentially increases the perception of brand fit for a given audience. With that change comes receptivity to other messages, in other media, until finally the sale is made. (N.B., we actually have measured advertising effectiveness for urinal ads, airport displays, interactive kiosks at malls and even truck stops–trust me, it’s going to be a long time before a social media enthusiast can spit out a form of non-traditional advertising as an example of an “unmeasured” media that isn’t actually being measured.)

Social media’s role in cross-marketing effectiveness measures has to be defined, measured and vigorously championed by social media consultants. We don’t yet have the metrics, perhaps (though I would suggest Brand Fit, Willingness to Consider and Net Promoter as promising candidates) but without rigorous, methodologically sound measures (not just audits!) of how social media affects consumer perception, social media will continue to be stuck in its silo instead of becoming a transformational force within the enterprise.

Finally, consider this. Twice in the past two weeks, I have heard the “social media ROI” question answered the same way: “well, medium ‘x’ isn’t measured very well either.” The answer is never to say “the other guys are just as bad.” The correct answer is always to innovate.

I continue to despair at the state of journalism when it comes to data like this. I am already seeing this crop up on sites as “19% of Americans [not even Internet users] Tweet!” A pocket calculator would tell you that 19% of Americans would be about 60 million people. There is plenty of clickstream data out there to make the point that this is a patently ridiculous estimate of actual Twitter users. Yet I will see 19% reported–and retweeted–all week long. But maybe it’s time for me to stop beating up on journalists so much, and look at the actual survey in question.

The last time I quibbled with the reporting of this data, I pointed out that most people actually update their status with Facebook, not Twitter, and showed the Compete.com traffic as a gross comparison. The gap between Facebook and Twitter continues to widen, if anything, making the wording of this particular question curious, indeed. While I will continue to highlight the improper reporting of stats like this, maybe it’s time to challenge the wording of this question itself. Asking Internet users if they use Twitter “or another service” to update their status is roughly akin to asking them if they consume “Sprite or another soft drink” and reporting the results as “Sprite Drinkers.” I realize that this would blow up the trending on this particular question, but if Pew themselves are going to continue to call this report “Twitter and Status Updating,” it would be helpful for the rest of us if they would break the question out into “Twitter status updating” and status updates on other services.

The resultant data would be less sensational, perhaps, but far more useful. It is inevitable, as status updating apps proliferate, that the role of Twitter in status updating will continue to shrink, not grow, while the Pew number goes up with each report. As I write this, I still have no idea how many unique Americans are actually using Twitter, and that in itself would make for a sensationalist headline, no?

I have to admit, the price actually intrigued me. While I still rely almost exclusively on Google, Wolfram Alpha is getting better at handling the more academic queries I have on a day-to-day basis, and the thought of a premium experience on the iPhone was briefly tempting. First, however, I wanted to see what the “free” experience would look like. I went to mobile Safari and accessed the Wolfram Alpha home page, and discovered that it was well-optimized for the iPhone. I then added the page to my iPhone home screen, and was rewarded with a nice spiffy icon. Looks almost like an app, doesn’t it? And it cost me nothing. Spending 50 bucks to get an app version of this sounds pretty silly, doesn’t it? I guess TechCrunch is right–must be a bad move.

Except, look what it made me do–it made me visit the page, use it, and add it to my iPhone home screen–where it previously didn’t exist. Plus, we are all talking about Wolfram Alpha again. Commenters all over the blogosphere are talking about how all you have to do is visit the home page on your iPhone and tap Add to Home Screen and you, too, can “outsmart” Wolfram Alpha by getting access to their valuable service for free. Silly Wolfram!

There can be an enormous gulf between “free” and “worthless.” Wolfram just implanted a $50 value in my head for their free service, and that seems pretty smart, to me.

I have been an avid supporter of the podcast industry for years now–I hope my body of work illustrates this–so I am pleased to see one of our leading podcasters blaze yet another new trail and further open doors for other content creators. I do have two niggling concerns, however.

First, there was a statement from Leo that we are no longer “new media,” we are THE media. That can be interpreted two ways. One way to read that statement is that we–traditional and new media–are now one, and we can put divisiveness behind us and collaborate to make great content (and $$$) together. The other way to read it is a bit more “us vs. them,” and as rousing a rah-rah speech as that might make at Blogworld, I truly hope we all don’t feel the need to play that game.

Second, I can’t wait to watch TWiT (and other niche content I am passionate about) on my good ole’ American McBigScreen. But consider this: when video podcasts were confined to portable devices and desktop computer distribution, they competed in the same space as Dramatic Squirrel and crying Britney girl. Now, podcasts will co-exist side-by side with CSI, Glee, Top Chef and (soon!) American Idol. I realize that this consideration set has technically been in existence for some time, but there are two barriers to entering a consumer’s considerations set: technical, and psychological. The technical barrier is breached. Getting consumers to place podcasts and broadcast television programming in the same mental consideration set is still an open question. Let the competition for my media time begin in earnest.

Setting aside the irony that we are celebrating the face that we can now watch new media on that big old media box, this is still awesome news, and Leo deserves a hearty slap on the back. Now, for everyone in the space, you’ve really got to up your game.

The argument against the Focus Group essentially states that better information–more observational, in situ data–is available by mining through the wealth of unstructured data now available in the data streams of various social networks.

There are compelling aspects to this argument–in particular, the belief that “tweeted” comments about brands are somehow more authentic (and certainly less artificial) than similar comments derived from qualitative research constructs. There is an undeniable truth to this–though I would argue that the very act of “Tweeting” to followers carries with it an implied and palpable Heisenberg effect for all who tweet–The Twitterer knows someone is listening, so the Tweet is subtly changed; unconsciously adapted for an audience the author may never know, but wants to please nonetheless.

Consider this, however. As seductive as all that unstructured data is, where do you think it came from?

The 11% of Americans who post status updates?

The 8% of Americans who contribute to blogs?

The 5-8% of Americans who post to Twitter (your guess is as good as mine here, given the number of duplicate/SPAM accounts)?

The 1 in 5 Tweets that are about brands?

The 1 in 5 Tweets about brands that actually express an opinion about those brands (i.e., really 1 in 25 tweets)?

When you consider that there is nearly 100% overlap between all of these groups, you are left with the inescapable conclusion that fewer than 10% of Americans are contributing unstructured data about brands, which leaves the other 90% essentially voiceless in this particular model. My neighbor, for instance, doesn’t post to Twitter, only uses Facebook to share family photos and eavesdrop on her kids, and certainly doesn’t blog. She is, however, a professor at Duke and drives a nicer car than me. Don’t you want to sell her some stuff too? The fact is the vast majority of Americans are online, but don’t post about brand experiences online, and going exclusively by the percentage that does share brand opinions might be useful in some ways, but might be horribly misleading for a company seeking to skate where the puck is going. For every PayPal there are 10 Flooz/Beenz-alikes, and calibrating the opinions we can aggregate online still benefits from an offline reality check.

I think the smartest thing one could possibly say about this is that in every focus group I’ve ever moderated, there are 2-3 vocal, opinionated peer leaders, 5-7 that will go along with the crowd in public, and 3-5 that won’t go along but won’t challenge the room. As a moderator, I can see and feel this palpably, and get to the heart of the true opinions in the room regardless of the articulation gap that may exist between respondents. If I mine unstructured data, I would only get the former–and worse, I wouldn’t really know how many of the latter groups existed. Sample is everything.

So, the non-response bias for “killing” the focus group and other research projects is enormous, and incalculable. Yet, the unstructured data that we can glean from social networks is potentially very valuable, and absolutely supplements and in some cases supplants observations from other forms of market research. Let’s not lose sight of that–I’m not–but let’s see it as adding incremental insight, and not as the sole source of consumer insight.

I think a better way to think through this is not to use zero-sum thinking. Social networking has enriched my life, and given this quirky introvert a whole new way to express himself before his friends, peers and even potential clients. But it didn’t replace the relationships I had before, or how I built and nurtured those relationships. It just made them richer.

Unstructured data makes consumer insight richer–appreciably so–and any market researcher worth their salt will use it. But let’s use it to make our focus groups richer, and our surveys more informed–not to exclude the reluctant majority who don’t contribute brand opinions online and may not have shared experiences with those who do. Let’s use them both, smartly, to create a substrate of data that can provide more actionable and useful market research than ever before. That’s what gets my juices flowing.

In any business venture or investment, you get what you measure. While some of the panelists seemed knowledgeable about potential measures of success, others were a bit laissez-faire about things like ROI. “Forget about ROI,” claimed one, “just do it!” Companies can forget about ROI, I suppose, when they have unlimited resources–but as long as there is an opportunity cost to pursuing social media initiatives, there needs to be a way to justify that cost. Another panelist claimed that other forms of advertising (like airport displays) didn’t have metrics either–but that’s patently untrue (I know this because we actually measure airport display effectiveness at Edison.) There are clear measures like Willingness to Consider and the Net Promoter metrics that are as applicable to social media as they are to any kind of sponsorship, out-of-home media or interactive display. I do a lot of work for Public Radio, and there is an enormous effort within public media to measure the “halo” effect that sponsorships have for the companies that sponsor programming. I suspect there are clear analogues from measuring how people feel about companies that support public media, to how people feel about companies that engage in social media. I’m talking about clear measures: metrics that equal dollars by quantifying the value of loyalty and engagement.

The metrics are there–it’s simply a matter of doing pre/post measures on the ones that matter and off you go. The smartest people I know in social media know this–but there continues to be this prevailing received wisdom amongst social media enthusiasts which insists that, because social media engagement is the right thing to do (I don’t dispute that) that we should screw metrics (which I clearly differ with.) Metrics justify investment, which raises the profile of social media within the company, which in turn attracts the notice of the functions within the company that truly need to understand the transformative power of social networks. And I’m not talking about the marketing department.

This brings me to the second problematic issue with social media “strategy.” I firmly believe that there is a strategic component to social media, and that it absolutely has the power to transform the way we do business. The problem that a lot of social media marketing consultants have, however, is that ultimately they are going through the wrong door. If an employee has an interaction with a disgruntled customer on Twitter, that interaction may fall under the aegis of the marketing department, and the successful resolution of that interaction probably will involve any and all of the tips espoused by the experts on the IMS09 panel. When Southwest Airlines resolves a baggage issue on the social web, or Ford engages online with potential Fiesta drivers, those are marketing initiatives, without doubt. Each successful interaction is a tactical interaction–person to person (with an audience, of course!) The “playbook” that a marketing employee follows in such an interaction may indeed originate from the marketing department. But the actual ability for a rank-and-file employee to be a real human, to stop the assembly line (like a Toyota factory employee can) and communicate transparently with customers and prospects, goes above the marketing department’s pay grade. At the very least, it is a Human Resources issue, and ideally a CEO-level intervention. Telling an internal social media marketing specialist to “be human” might come from the marketing department, but only a company that has been engineered from the ground up to support a culture of human business has the ability to empower that employee to actually be human.

I think Chris Brogan gets this–he articulated it today in an article entitled What Human Business and the Social Web Are About. Brogan notes that engaging in social media “is not a new marketing channel [or a] new technology…it’s more. This isn’t the battle of who “gets it” and who doesn’t.” I think he’s right. Yet you read through the comments to this post and you still see a lot of language around people “getting it” or “not getting it.” I don’t think anyone “gets it” (myself included) because we don’t know where this is going. Telling a marketing department to be “helpful and human” on the social web is a tactical message. Reengineering the company so that its employees can actually be helpful and human may require an enormous overhaul of the very theory of the firm.

What I think Brogan is recognizing is that it’s time for Act Two for social media. We’ve heard all the admonitions from social media gurus and enthusiasts about how we are meant to act and behave on Twitter. But going through the marketing door is only going to get you so far–in that sense, social media helps the rich get richer, but does little to help companies that aren’t structured to actually be transparent and helpful where the rubber meets the road. Social media’s next act has to be through the corporate HR director’s door, and ultimately the CEO’s door. To get there, however, social media has to prove itself by more readily embracing effectiveness metrics (not just statistics) to raise its profile in the organization. Give the CEO the numbers to justify continued social media engagement, and that engagement will continue–and ultimately be the trojan horse for what human business can become.

There are 5 basic questions any consumer–and any reporter–of survey data should always know the answers to before attempting to process the significance of a given survey research project:

1. Who Paid For The Survey?
2. How Were The Questions Asked?
3. Who Was Asked?
4. Who Wasn’t Asked?
5. What Was The Exact Question?

Survey research projects that adhere to AAPOR standards must be clear about all of the above (with #4 inferred from #3, of course). Today’s case in point is with a recent survey of Twitter users in the Crowd Science network. I would encourage anyone interested in social media data to read the actual report (which is fine) and not the “reporting” of the report, which varies wildly in quality. Exhibit A: MediaPost’s Research Brief on the study. Here’s an excerpt:

Additional survey results include:

41% of Twitter users prefer to contact friends via social media rather than telephone, compared with 25% of non-Twitter social media users,
11%, vs. only 6% of those not using Twitter, actually prefer social media over face-to-face contacts

14% of Twitter users said they have revealed things about themselves in social media that they wouldn’t under any other circumstances

8% admitted to “frequently stretching” the truth about themselves online
Twitter users tend to be older than non-Twitter social media users (54% over 30 years old, vs. 42%),

They are twice as likely to be self-employed or entrepreneurs (18% vs. 9%)

24% vs. 15% “buy gadgets/devices when they first come out,”

48% vs. 30% have created a website

37% currently maintain a blog, twice as many as non-Twitter social media users

The Crowd Science study was conducted across more than 600,000 visitors to multiple websites between August 5-13, 2009, targeting social media users age 12 and up.

The issue I have with most social media data like this is not with the study itself, but in how the study is reported. For instance–if you read the study, you will know that the actual sample for these statistics was not 600,000, but 718. The total Crowd Science sample base for their social media data may be 600,000, but the number of Twitter users who responded to this particular survey was 718. My point here is not to point out that the survey had a small sample; rather, it’s to point out that there are some gaps in how the survey was reported. We’ll just call that a sin of omission–it’s not really what I want to dwell on here.

The most egregious violation is far more subtle–the term “Twitter users.” Twitter users is sprinkled liberally throughout the MediaPost story, and saying that Twitter users do this or do that makes for compelling–and tweetable–sound bites. However, it’s simply incorrect to say “Twitter users” unless you are sure that you have a methodologically sound, representative sample of Twitter users. Which, in the case of this particular study, is not the case. If Crowd Science’s report claimed this, then the fault would lie with them–but they don’t! If you actually take the time to read the whole report, you’ll note that Crowd Science was methodical and careful in their description of the sample as either “respondents to this study” or “Twitter users in this survey/we surveyed,” etc. When a survey project uses a sample of convenience, as this one does, then the reporting is always on solid ground by referring to the sample as “respondents to this study (who use Twitter). Referring to them as Twitter users, however, implies that the results are projectable across the universe of Twitter users, which cannot be claimed in this case–and, to be fair to Crowd Science, was not claimed in the report.

The methodology of the study was fine–Crowd Science isolated the 718 Twitter users in a larger pool of visitors to websites in the Crowd Science network. That’s who was asked (see #3, above). Who wasn’t asked? Twitter users who were not visitors to Crowd Science network sites, for one, and we cannot easily characterize the non-response bias there without significantly more data about the Crowd Science network. Again, please do NOT take this as a knock on Crowd Science or their methodology–these concerns are common to many self-selected online surveys. The issue here is the journalistic shorthand that converts “Respondents in this study who use Twitter” into “Twitter users.” It’s the latter term that gets retweeted ad infinitum into everyone’s 140-character summations of this survey, and represents the real crime of this sort of reporting.

What this sort of sloppy reporting does is create a universe where one report of “Twitter users” from survey A contradicts another report of “Twitter users” from survey B, which gives marketers fits and reduces confidence in survey research, even though surveys A and B were both probably perfectly fine.

Got questions or observations about social media data? I’d love to hear from you.

However, looking at the quintiles Penn State used to characterize sentiment, I’m not sure that’s entirely accurate. Brand researchers generally only look at the top two boxes (and sometimes only the top box) as truly positive measures of sentiment, and in this study 33% of the tweets were labeled “Great,” while 19% were “Swell.” The other three categories ranged from Wretched to Bad to So-So: the latter would not generally be seen as necessarily “positive.” In any case, about 52% of sentiment was positive, and 48% were “non-positive,” if not necessarily negative.

On one hand, this means that brand tweets are not overwhelmingly positive, but they aren’t overwhelmingly negative either. And with most brand tweets not expressing any opinion whatsoever, the whole thing looks, well, kind of “normal.” All of which makes me wonder if the Twitter userbase is slowly lurching towards the center. Clearly it isn’t there yet, with the majority of the noise generated on Twitter by a disproportionately small number of users, but observing that the Twittersphere appears to be at least well distributed in terms of sentiment gives social media data miners and brand managers some comfort that their products and services won’t necessarily be savaged on Twitter, and in fact most brand tweets are either expressions of usage or exchanges of simple information–all of which are entirely appropriate fields in which brand managers could (and should) engage.