In the same way that YouTube spawned a surge of new filmmakers, Tableau Public — free, running the same engine as its desktop sibling, and embedable — will bring on a new generation of data players and spectators.

I was a spectator at a data visualization conference one afternoon two years ago. Tableau Software director of visual analysis Jock Mackinlay had finished his presentation and another person had started his. Yet someone at the control board forgot to flip a switch, and Jock’s live screen remained on one of the room’s big screens. Jock assumed his screen had been hidden, and he kept playing with the data. I don’t have to tell you who seemed to have the audience’s attention until someone pointed out the problem.

The mere visual distraction was minor. Even without narration, I got caught up in the apparent drama as he tried one look at the data after another.

Not long after that, I wondered aloud to someone at Tableau about data hobbyists. I imagined people who foraged for data to analyze then publicize it to start conversations, collaboration, or duels. Data would be their raw material of choice just as scrap metal is to some sculptors or overheard conversations is to some fiction writers.

There was no such community visible then. But I realized this week that I know one now: Dan Murray, a skilled, dedicated Tableau user. He jokes that he’s a “freak” because he analyzes data from the federal budget and posts his often provocative analyses. He’s already been answered by at least one who disagrees with him.

In beta and since its February 11 launch, Tableau Public has hosted a flurry of visualizations, including these: a map of top venture capital firms investments by U.S. region; a chart showing how long it takes to build a technology empire; a history of earthquakes in Haiti; a neighborhood breakdown of housing supply in Seattle; trends in U.S. high school graduation; and studies of deprivation and marginalization in education. In most cases, spectators can become players by selecting subsets of the data to find answers to their own questions.

With popularity comes some misuse. Many of the charts will break rules, such as what happens in another kind of game, YouTube. A New York film editor I know complains that many YouTube-acculturated film editors have neglected basic editing principles. She writes that they rely so much on special effects that they “can’t put two shots together and have them work as an unembellished edit.” On Tableau Public, there will be pie charts, chart junk, and even baselines that do not start at zero. We’ll survive it.

But what’s all this got to do with the very serious practice of business intelligence?

Like monks must have done when printing presses began producing books for the masses, many priests of business intelligence will stand aside, arms folded in the aspe chapel. But I predict that before long even they will appreciate a wider, deeper pool of analytical talent ripening for training and employment.

I suspect that the new bunch will have been sharpened by the give and take of public exposition. They’ll also learn from playing in a huge community the way artists and craftspeople of all kinds improve their skills when they bump into peers every day.

This is a new clue for the future of BI. It can’t help but improve data analysis in business. So let the games begin.

How about sales people who push federation tools on those who actually need data warehouses?

“It all sounds extremely sexy,” says my source, who works for a major business intelligence vendor and whom I can’t identify. “You have a lot of people who exaggerate their ability to combine data to provide business solutions. … They don’t prototype, they don’t profile, they don’t actually think about the problem or do testing or even send some high school data analyst out with Excel to put something together that [the customer] might want. They don’t do that.”

Many sales people tout EII because that’s what they have to sell, he says. “The EII tools give you your data, warts and all,” he says. It’ll work fine as a data warehouse substitute “if the data’s pretty clean to start with, if it has a somewhat similar structure, if you can define the data you need, if the data’s relatively common across all the sources, and if there’s not much duplication.”

Even if the salesperson has a more appropriate tool than what the customer asks for, the customer may never hear about it. “‘Fine!,’” thinks the salesperson. “‘If you want to buy a hammer, that’s fine. If you want to buy a wrench, that’s fine. It’s not like I care. It’s just sales to me.’”

Just once, says my source, he’d like to hear one of these questions: “How long does it take for a novice to become OK at this task?” Or, “How long would it take for an expert to become proficient at these two things?” Or, “If I have a failure, what is your tool’s usual process for recovery, and what gives your tool more integrity than others?”

Mark Madsen, meanwhile, has been been thinking about similar problems but from a different perspective. He’s research director at the Third Nature consultancy and a keynote speaker at this month’s TDWI conference in Las Vegas.

One source of problems he sees is vendor marketing. “It’s all about ‘our tool does this’ or ‘has these features,’” he writes in email. “A lot of people don’t think about them that way. They think about them as ‘what this tool is for.’” People end up using an ETL tool for real-time synchronization, for example, or a federation tool in place of a data warehouse.

Even product documentation can lead users down dark paths. “All those docs that say what the features are help when you know what feature you want,” he writes. “When you’re trying to accomplish a task, you’re thinking in a different way.” A common result: convoluted solutions.

“I once did something in an ETL tool,” he writes, “and the product developer said, ‘That’s not how you do that.’ They had built around an improper conception of how users apply it.”

Design schools tell you that every user has a theory of how anything works, he writes, which determines their approach to it. Wrong theories explain why people push on doors that need to be pulled, for example. He says that this insight has made him change his approach to teaching his courses or showing clients.

“I’ve realized that I need to start with the ‘what this thing is for’ and move into what you do with it, and how it works.”

Mark may go into this more in his keynote at this month’s TDWI World Conference in Las Vegas. His long-running “Clues to the Future of Business Intelligence” — perhaps the “Cats” of tech presentations — has been one of the most interesting I’ve seen in any tech industry. I expect “Stop Paving the Cowpath” to be worthwhile.

“I like Vegas. Should I go?” he asked from somewhere that sounded far away. I said it all depended on what he wanted to learn. Is making sense of his data important? If yes, go. But there seemed to be more to his question.

He’s gone through one shady business after another since the early ’60s, when as a teenager he sold drugs on the street. Now he sells fake email addresses in huge blocks to Eastern European spammers. All his customers have had a good education, he tells me, yet most retain some of their families’ traditional ways. He describes them picking over his blocks of email addresses as if over oranges in a bin, rejecting one, taking another. They seem to rely entirely on feel, and Marco makes sure each new batch feels “fresh” and authentic year after year.

“Cool. My data’s real, real important to me,” said Marco. “So’s my know-how, my experiments, my research. Those experts in Vegas dish on how to manage all that, man?”

Definitely the data, I said, but not much on the qualitative end of his research. He was disappointed. 

“You know, you got me going on this insight thing, man,” he said. “And then you change the story. This business intelligence takes care of only some of my insight? Only some of it? What do they think, data’s the only way you get insight?”

He had a point. I thought fast. I said he should think of his operation like a speeding car. He liked that. I said he needed a “dashboard” to let him know how he was doing. He liked that, too. There was a course on Tuesday, I said, all about that.

“Cool, man. But what about my research? I got these journals I keep with my results and theories and shit like that. What about all that? I keep losing track of it all.”

I said I thought he was talking about knowledge management or something. 

“Yeah, that sounds like what I want. Knowledge management. They don’t do that there?” 

I explained that data was this event’s main focus. Other events … but he cut me off.

“No, man. Here’s what it’s about,” he said. “It’s all about marketing. I don’t know much about business intelligence, but I bet that every benefit, feature, whatever comes from a different tool. Each comes from a different vendor,” he said in a tired sing-song, “and the producers of this event have a line on a certain kind of vendor. To protect their game, they make up a category. Get hip, man. It’s always like that.” 

He quickly added, as if he had already bored himself, “How’s the food there? Can a guy score somethin’ to eat?”

The best Caesar’s can offer, I said. Then he had to go answer the door. I heard urgent knocking.

When Wayne points to a trend, it’s news even if others might have already foreseen it. He’s one of the industry’s most thoughtful observers, and one of the most deliberate.

In Tuesday morning’s blog post, he suggests improving BI by enlisting those who study how people make decisions.

To take BI to the next level, we need better insights into human behavior and perception. In other words, it’s time to recruit psychologists onto our BI teams.

He gave an example of one place that could have benefited from visits to the shrink’s couch.

A recent article in the Boston Globe called “Think Different, CIA” provides some instructive lessons for companies using BI tools to make decisions. The article describes a phenomenon that psychologists call “premature cognitive closure” to explain how humans in general, and intelligence analysts in particular, can get trapped by false assumptions, which can lead to massive intelligence failures. It turns out that humans over the course of eons have become great at filtering lots of data quickly to make sense of a situation. Unfortunately, those filters often blind us to additional evidence — or its absence — that would disprove our initial judgment or “theory.” In other words, humans rush to judgment and are blinded by biases. Of course, we all know this, but rarely do organizations implement policies and procedures to safeguard against such behaviors and prevent people from making poor decisions.

See his full post here.

Be sure to see the comments, too. He writes in reply to questions, “Like data governance, we need some principles for approaching and managing decisions. Maybe we should start a decision governance institute!?”

I can’t help notice: an institute.

See “CIA’s insights on the psychology of analysis” on Datadoodle.

See his latest, posted yesterday afternoon, on Gary Klein’s Streetlights and Shadows.

About 50 million people play Zynga games every day. It’s the leading online social gaming platform, according to Ken, and it’s grown from zero in 2007 to revenues of “a few” hundred million dollars annual revenue. Every day, the company captures 20 to 30 billion records of data, and Ken and his team use that data to improve revenue, viral marketing — and customer retention.

Zynga players play free. The revenue comes in a few dollars at a time for “virtual goods.” In the popular game FarmVille, for example, a player might get tired of the old-fashioned plow. The tractor upgrade costs $2.

“There are tons of different ways you can think about retention,” he laughs, “and which one should we use?”

How do you know when a customer has left? “Unless we don’t get a note saying, ‘Hi, we’re no longer playing,’ how do we know?”

Of course, no player’s going to make it that easy, so how long should Zynga wait before considering the player gone? A week? A man could have dropped his virtual pitchfork for a real vacation — or he could have plowed the last row.

Ken dealt with analytics all the time at LucidEra, but games were new to him. He’s learned a few things.

“It turns out, as you might imagine, that it depends on the game,” he says. The average simulation-game player tends to visit frequently, for example. Poker players, though, are much more likely to come back after, say, a three-month gap.

The retention curve also varies by the length of each player’s tenure. A new player who stays away 30 days is much less likely to return than a player who’s been at Zynga for years. Ken now puts users in three basic tenure buckets: “new,” “mature,” and “elder.”

Whatever question you try to answer, it has to be actionable. “There are metrics, and there are metrics that matter,” he says. If volume plunges, were the missing players mostly new ones? If so, it could indicate frustration; perhaps the games need better tutorials or less functionality at the beginning. Or were most of the missing the long-term customers? If so, perhaps the games haven’t offered enough challenge.

Ken expects growth when the economy improves. “When we look at what happens over holidays, such as July Fourth and Thanksgiving, usage really drops. Then it picks up as people go back to work,” he says. “[The games] are part of their routine. On vacation, players break their routines. They sleep late and spend more time with family. They don’t play the game.”

“It’s fascinating,” says Ken. “In analytics, so much of the problem is figuring out what the question really is.”

I think he means that it’s a great game.

“Who should get fired?” is the same conversation as after screwups in corporations, writes Dychè, principal at Baseline Consulting.

Instead, the government should be addressing process issues. Indeed, the real conversation should be how to move forward. These questions should be asked now: “How should we bring identifying data together? What are the key sources? How should integration, access, and usage policies be formulated? What would a sustainable process look like?” Those questions aren’t “who” questions, they’re “how” questions, and they should be front-and-center in the national security conversation.

Read the full blog post.

A culture failure is much more alarming and also much more uncomfortable than a simple process or technology failure. It signals that something is fundamentally wrong, something which is very complex and hard to change. It means that you not only have to change your own attitudes and behaviors, but also those of your colleagues, including management. You might need to change the entire incentive model, which in the end determines the bonus of your CEO. What is worse, you most likely also need to change the attitudes and behaviors of your CEO (”Impossible!”).

Originally referred by Henrik Mårtensson on Twitter.

Social media lets us try for a kind of politics that was impossible until now. I hope for great results. For many, Government 2.0, or “collaborative government,” will mean just “friending” a local cop. But in full flower, Government 2.0 can mean far better service, and far more government-and-citizen collaboration than ever before.

Even before we had social media, the glare of public attention was a proven antidote for bad politics. Citizens getting up their elbows in policymaking has always been another strong medicine.

Trouble is, that “sewer-dwelling raccoon” is always smarter than people think. When he’s hungry, he purrs like a housecat and covers stinky laws with high-minded names. Advertising fools just enough voters — so often complacent and impatient — to throw a new law onto the books. On and on it goes.

Such a stinky new law is just what Californians got in 2000. Proposition 34 was sold to voters as campaign finance reform. It was a ruse. (A few days ago, a report confirmed suspicions, and a major drafter of the proposition insisted he was “outraged.” Yeah, and round up the usual suspects.)

One other fix, more honest, came 100 years ago: California amended its constitution to give citizens the ballot proposition. It was the only way for voters to bypass the paralyzed Legislature and loosen the Southern Pacific Railroad’s grip. It worked. But more recently, ballot propositions have helped tie the state’s budget in knots.

In the long run, who knows how social media, visual analysis, and other tools may be used in government? What will matter most of all is who uses them. If it’s “the people,” which people?

I hope this new, pervasive politics mobilizes a new wave of smart activists — the way desktop publishing and, later, weblogs enabled editors and writers. Or the way tools like Tableau and Lyza are enabling independent-minded, creative analysts today.

As these activists learn about politics, I also hope that more citizens than ever before step up to watch, push, and verify. Such a voter would be Citizen 2.0, the real hope.

Otherwise, it’s going to be that raccoon again — this time on Twitter.

If you try to become more cost effective by reducing capacity, and thereby capacity cost, all will be well at first, from an economic point of view. (The people who are let go are usually of a different opinion.) The catch is that this will increase the queues in the system. This increases lead times. Consequently, cost associated with lead time will also increase. (In manufacturing there is also a considerable storage cost due to increased inventory, but we will ignore that for the purposes of this blog post.)