Scratch the surface of most presentations and conversations — such as last week at the TDWI conference in San Diego — and you find people problems bobbing right up alongside data problems: indifferent executives who undermine BI, short-sighted silo keepers, and IT people who enrage business users with paternalism, to name a few top quirks. If only data were all we had to transform!

One business manger at last week’s TDWI conference in San Diego told me that one of his most daunting tasks during a recent data warehouse implementation was persuading silo managers to release their death grip. For this task, he was on his own. Couldn’t someone have briefed him on the objections he was likely to hear? Or tactics to overcome resistance?

One organization that seems to have solved its people-problem was Blue Cross Blue Shield of Kansas City. Their impressive success with Hewlett-Packard tools was based on commitment to data for strategic advantage and shrewd orchestration. They also had a steady, guiding hand from HP. For example, as Blue Cross Blue Shield built new structures, it avoided upsetting stakeholders by leaving old structures in place for 18 months. (I hope to have much more on that story in the next couple of weeks, thanks to John Santaferraro, HP senior director of marketing, business intelligence.)

Several people in the BI crowd do talk often and thoughtfully about organizational problems. Maureen Clarry, CEO of CONNECT: The Knowledge Network and longtime TDWI instructor, teaches “Power, Politics, and Partnership in Business Intelligence Projects” at every TDWI conference. Participants see for themselves how position shapes behavior. Those short-sighted silo keepers, for example, could flip into data-sharing maniacs if assigned a different position.

Jill Dychè, partner at Baseline Consulting, teaches “BI from Both Sides: Aligning Business and IT,” with strategies to avoid or pave over organizational potholes. She suggests, for example, dodging the perception that BI is “so much data loading and report provisioning.” She writes in email, “We find that the extent to which BI is viewed as a program — with platforms and tools merely components — is the extent to which BI teams are productive and visible in their companies.”

Wayne Eckerson, director of TDWI Research, also addresses these issues, most colorfully with his idea about “purple people.” They are a little bit business-blue and a little bit technology-red, and the purple coloration they acquire lets them traverse the IT-business rivalry.

Wayne spells out some important characteristics for this job, such as maturity and knowledge of technology and business domains. The best are “switch hitters,” by which he probably means to imply that they’re persuasive wherever they stand. In fact, “purple” sounds like a euphemism for another “P” word that Jill actually spells out: politician.

Bad word or not, it’s a critical function. A good politician’s essential function is to coax rivalrous parties into agreement. If that’s the kind of function Wayne sees for the purple people, then they really are, as he says, “the key to BI success” — at least at one level.

Purple may not help much at higher levels. Wayne’s knowledge of of business intelligence is far deeper than mine, but my experience elsewhere makes me think these people are just one of many keys. When I was a sort of purple person myself — in the late ‘90s, bridging an arrogant Web development group and a couple of marketing groups accustomed to full control of their media — my own skill at listening, negotiating, and arm-twisting was only one key. Another key was my boss. At first I had a strong one, later I had an indifferent one, and even later I had virtually no boss at all. I felt like my district shifted boundaries each time, my agenda with it.

One friendly executive suggested I stand up and promote the Web project around the company at any meeting that would let me. He said, “Show ‘em how great it is, and the credit will rub off on you.” Just like a politician running for office.

If I were a purple person today working in BI, where would I go after I’d exhausted training by Maureen, Jill, and Wayne? Most likely, I’d turn for inspiration to books on politics and influence, such as biographies by Robert Caro. Actually, I’ve gone there already, but only because to me politics is a good word. No, you don’t want to emulate Caro’s subjects, just clean and adapt some of the principles they used.

One thing seems clear to me: If purple people, would-be purple people, red people, and blue people are to expand the BI horizon, conversations have to go longer and deeper into the people problems. We start by ending the prissy avoidance of that word that at its best connotes people, perceptions, and compromises: politics!

More than 30,000 visualizations — “vizes” — have been published. The most popular of all, says Elissa, have been the ones about homes, personal budgets, and leisure. One of her own favorites is a local real estate blog, Seattle Bubble. “I wish I could have seen blogger Tim Ellis’s data in Tableau Public before I bought my house.”

Another favorite of Tableau staff, who are said to have a healthy contingent of foodies among them, is about cows and their milk. Vizzer Kate Golden at Wisconsin Watch charted the number of cows over the last 80 years in Wisconsin with the gallons they produced. Dairy farmers have 47 percent fewer cows today than at the peak in 1944 and ’45, and they squeeze three times more milk out of the cows they do have. In a YouTube-like moment, vizzer Carpe Diem responded. He mashed in milk prices. They’ve fallen, though it’s unclear how much; the viz fails to note whether the prices are adjusted for inflation.

The visit count keeps accelerating. Past growth feeds more growth. The big names that have joined in help, such as USA Today, The Seattle Times, and CNN Money. There are also influential bloggers like Mish’s Economic Blog and Infectious Greed also pull in visits. But the highest growth rate is among sports bloggers, such as pistonpowered.com and school bloggers like Gothamschools.org.

The “beef” — as in “where’s the beef?” — is whether Tableau Public really is becoming the YouTube of data? It seems to be on the way there.

The crucial factor that distinguishes the YouTube from the NotYouTube is the network effect. The genuine YouTube is the default, the unquestioned center stage. An also-ran may have faster servers, nicer staff, and more permissive rules, but it’s still not YouTube. With volume like this, Tableau Public is well on its way to becoming a true YouTube.

In the meantime, there seems to be another reason for satisfaction at Tableau Software. Elissa asserts “plenty of evidence” that new, purchased licenses for Tableau Desktop and Tableau Server are coming in that started with awareness of Tableau Public.

In the most recent example, a simple Tableau Public-hosted chart on Wired caught me looking. And thinking. It hooked me with a bar chart that compared rates that iPad users pay for downloading data.

Did I say that I really don’t care what iPad users pay per gigabyte?

I looked. There’s the U.S., I thought when I saw it, losing again. I’m used to that by now. But who’s losing worse? Belgium! Why Belgium? I thought of reasons, but none seemed to explain why a gigabyte there was more expensive than a gigabyte in Italy, another country I know a little bit about. Better price supports for waffles than for cannoli? My local Belgian thinks she knows the reason: “Taxes!,” she says. But I told her that that explains nothing at all, and that conversation continues even today.

Sure, it’s all a lot of fun. It’s powerful, too. The simple chart that can hook you on the fly — by making you notice, making you scratch your head, making you try out one angle after another in the quiet of your moment’s pause — can hook customers with modest budgets and legions of casual users to excite.

Repeating the warning, though, makes you look like a party-pooper. In 2007 at the TDWI conference in Las Vegas, a keynote speaker raised it one morning. Jonathan Koomey — author of Turning Numbers into Knowledge and one of those voices the BI world needs more of — did his best. But I could see the unfolding disaster from my banquet table, as attendees glanced at each other in scorn. When the lights went up, not one person raised a hand with any question or comment.

Now Sex, Drugs, and Body Counts: The Politics of Numbers in Global Crime and Conflict, edited by Peter Andreas and Kelly M. Greenhill, tries it again.

You may wonder what sex, body counts, and politics have to do with data analysis, but try to keep an open mind here. The book promises to let us spit out the usual cud of business intelligence, data quality, and get to the real spice: the politics of data. I can’t wait to read it. For now, see Jack Shafer’s review on Slate.

I won’t be surprised if the book points out how each organization’s core group subtly chooses the stories its data tells. I’ve just finished Art Kleiner’s Who Really Matters, which goes into detail on these groups’ formation and influence, including how they define who’s in, who’s out, and why. It’s the essence of politics.

Though core-group members may not ever lay their smooth palms on any data, data is nonetheless coiffed to suit these people. Through layers of managerial interpretation and re-interpretation, their influence cascades all the way down to tiny decisions about how data’s summarized, what’s measured, how it’s measured, and who measures it.

Like other forms of expression within an organization — speech, email, jargon, attire, hair style, suit or T-shirt — data is part of the politics. Though this has a big effect on decision making, it seems rare that I find it on a BI-event agenda. BI’s scope needs to widen.

Sandy had been busy analyzing data. For years on Wall Street, he pored over mortgage-backed securities with a tool he and peers developed for themselves.

He only learned of ETL recently. He’d become acquainted with a data architect with whom he shared a bus ride every day to and from their offices in downtown Manhattan. “I had never really spoken to him before,” Sandy recalls. “He was in a different world even though we both dealt with data.”

Sandy described to him his rapidly maturing tool. As I imagine the scene, the calm data architect suddenly twisted himself on the cramped bus seat to face Sandy. “You don’t do ETL? You work with raw data??”

No, he didn’t do any ETL, Sandy explained. “We didn’t realize how important that was,” he recalled. “We had always just stuck the raw data into the database and then realized, ‘Hey, this data’s a mess.’” He instructed users to clean it themselves. “You get the data from the horse’s mouth. You’re the expert. We didn’t realize how powerful this was.”

In Sandy’s system, you don’t worry about database design. He and his partners not only didn’t worry about ETL, they wondered how data analysis could not be done their way — import first, clean later. “It makes good sense if you can get away with it.”

A crucial factor that lets the tool work as it does is speed. It allows the 1010Data engine to calculate and recalculate repeatedly. The summaries that cubes harbor for anticipated queries are no longer necessary. Parallel processing with a columnar database runs fast enough. In place of ETL, he uses what he now calls “ELTAR,” for extract, load, and transform as required.

A hurdle, he says, is conventional beliefs held by his sales prospects. In one phone call recently, he explained to a prospect that ETL was unnecessary. The man replied, “That’s not credible.” In fine sales form, Sandy said, “Then you’ll be impressed when I prove it to you.” The prospect replied more firmly, “You don’t understand. That’s not credible.”

Actually, the technology’s credibility doesn’t matter much. The company, 1010Data, offers reporting and analytics on the cloud — invisible to customers except for the results. Sandy says, “We could have monkeys writing on scratchpads.” To those willing to try, he offers to prove it with the prospect’s own data.

Their technology’s speed allows them to do the work of dozens with a team of a few people, he says, and to finish large data warehouse projects in weeks that would otherwise take months or years. If multiple customers use the same data, such as stock market data, the time required is even less.

All without ETL.

Consider this description by a BI analyst: “Tableau provides business analysts speed of thought visual analysis on data held in memory on their desktop machines.” All that’s fine, but it may as well have been about a whole bunch of other tools, too.

At the root of this fuzz, explained McDaniel, is that most analysts who concern themselves with tools don’t actually use the tools. They rely on demos , marketing, and hearsay.

Though much of McDaniel’s recent work has centered on Tableau — his second book is Rapid Graphs with Tableau Software and he gives training sessions around the country — he also has a long, credible trail back through BI and data mining. He was director of analytics at Netflix, and has worked with more than 50 companies in BI. His first book was SAS for Dummies.

“I love SAS,” he says. Still, he remembers his sister in law’s reaction to his book on SAS. She was not an analyst but a “people manager.” These are the ones, he says, who have hated BI because “it had been made into a priesthood.” When she had looked through the book, she said, “Oh, this is great” and put it down. But she read the Tableau book for a half hour and said, “You should come talk to some people I work with.” She had recognized what she could do with the tool.

McDaniel’s sister in law and many like her don’t care whether the data is “in memory,” they don’t see themselves as business analysts, they take “desktop” for granted, and they know “speed of thought” is just gloss.

The list of features really doesn’t matter. All that really matters is whether someone can do what needs to be done with the tool.

McDaniel imagines a throw down, a data analysis match. It would be open to any BI vendor. Each vendor would send their best people, and each team would receive a uniform set of data. Over some defined period, teams would analyze and then present the results to a panel of vendor-neutral judges.

The reward? Perhaps a signed copy of a Stephen McDaniel book, or maybe a beer, possibly both. But certainly, repute.

What do you think of the face-off idea? Please write a comment.

A book from 2000, The Social Life of Information by John Seely Brown and Paul Duguid, suggests that BI designers have neglected basic human needs. Jack Vinson, of Knowledge Jolt with Jack fame, has just posted a worthwhile review that sent me scurrying over to Amazon.

Failure begins early for many new, supposedly revolutionary information systems. Designers “assume that the way people operate with respect to information has to do with only the information. … But there is a social life that revolves around the information that is much harder to capture and codify,” Vinson writes. “We look to verbal and physical queues for validity of what someone is saying. Our business processes have much more than just the inputs and outputs.”

Jumping forward but on the same thread:

… in the essay on reengineering … the authors describe how all the social life around business process is downplayed and often treated as waste. Businesses were re-engineered to remove much of the social lubricant that helped business flow. The essay on knowledge management was hopeful that KM would be a shift away from the intense focus on information and account for the human aspects of knowledge: that knowledge requires a knower. They have a great phrasing: information can easily be written down and transferred. But it is much harder to detach (and transfer) knowledge from the know-er and the context in which that knowledge resides.

The book is still important even after 10 years. It doesn’t even mention business intelligence, yet it addresses some of its fundamental problems.

Take a look at The Social Life of Information on Google Books. I also recommend Knowledge Jolt with Jack. Always worthwhile.

Many of the objections were of the “if man were meant to fly, he’d have wings” variety. But many others were valid.

The practice will run over a few bumps before it joins mainstream performance management and business intelligence. Unlike the impersonal data we know and love, keeping data about oneself can be uncomfortable, difficult, and downright weird.

One of the articulate skeptics called it “robot envy.” In his weblog, Marginal Utility, Rob Horning summed up his objections in the final paragraph.

Numbers can provide only one sort of “truth” about ourselves, and to pursue it we must surrender or compromise other kinds of truth—for example, the intuitive faith we have in our qualitative assessments of our dasein. [...] In other words, we give up our soul for a spreadsheet.

I’d like to meet the spreadsheet that steals souls. Until then, I’ll cling to my belief that no spreadsheet, not even Excel, has any more power to do that than a blood pressure cuff or a bathroom scale.

A more credible response came on the New York Times site from “Matt” in California.

Self-tracking will undoubtedly be used to oppress. It will wend its way into mainstream culture, eventually becoming something that employers expect of you as a matter of course. The temporal “productivity gaps” which we use to daydream, think about politics or other non-work related ideas, or simply consolidate memories, will be targeted and eliminated. Also, it is almost inconceivable that self-tracking data will avoid eventually going public.

Wolf gave his own response to some of the criticism (apparently a few minutes before Matt gave his).

I think many of the critical reactions make sense. What are we doing to ourselves? But I suspect that even the people who say something like “turn off the computer and go outside” are more deeply involved in the culture of self-tracking than they realize, and would benefit from going beyond initial revulsion. We _are_ in the process of changing. Our new selves will have new capacities as well as new vulnerabilities. Literacy itself was once a threat to our humanity: it interfered with memory, and substituted external representation for interior experience. It replaced living dialog with marks on a page. But we found a new sort of humanity in this world of letters.

The easy answer is that self tracking has to be done in moderation. Assuming it catches on, we’ll see public-service posters on buses and trains warning against overtracking and out-of-control “self love.” But every good thing is overdone and always will be. — and the solution has never been to ban it, deny it, or belittle it. It’s here, it’s coming, and we might as well use it.

See the article here, the 59 reader-recommended responses here, and all 138 online responses here. See the 7 letters here.

As you may know, the KnowledgeSync tool monitors activity and generates alerts. Event A occurs and, bang, an alert flies off by email, text, perhaps even ICBM. New sales inquiry? The tool can even issue a reply.

What could SAC want to do with such automatic alerts? Don, who is VP of sales and marketing, recalls saying to his boss, “‘You can take this customer if you want to, but do you really want to take that support call?’” Nope.

He and I went on to fondly recall those two 1962 films, “Dr. Strangelove” and “Failsafe.”

Tracking your time then and now is personal, it’s messy, and it’s the essence of business intelligence: collecting data and reading it for guidance in business activities that matter. Is there anything that matters more to an organization than productivity of its people? For a small office or home-based business, this might be the best BI there is.

This gets no recognition in the BI industry that I can find, at least not in the conservative world of TDWI. At least not yet.

PI — for “private intelligence” — has different issues, starting with data collection. In BI, data comes from transactions, all recorded routinely. In PI, most of it has to come from a “secretary” or from our own, tedious notation.

I dabbled in it once. The insights were good, if painful, but mostly it was tedious. A few years ago, a confluence of personal events let me do something I’d always wanted to try: hole up for a few months in a Sicilian village I knew slightly. The food was good, I had relatives nearby, and the nearby church bells rang all day and all night, four times an hour. At the same time, I had a book to edit. To stay productive, I made a game out of the work, tracking my time to the minute in Filemaker.

I liked the local food and started to hate the book, an office manual that inadvertently revealed a con game. Even so, I threw myself at it every day. But no matter how hard I tried, no full day ever resulted in more than about two hours of actual, productive work. My “quick breaks” for walks and coffee with a friend actually took up more time.

I made a Filemaker database because I could find no off-the-shelf product that would do anything close. Each period, no matter how short, had a starting and ending times I entered with buttons, and a calculation field figured the duration. A drop-down menu offered my usual activites. I could make a report for any period.

I thought some product would do that better, but I could find nothing. Then the May 2 issue of the New York Times Magazine ran an article by Gary Wolf about this, “The Data Driven Life.” My Filemaker invention wasn’t too far from what others have used, and now new devices are coming along that could make all that seem so old hat. Some people are even sharing their data on the cloud.

But as in traditional BI, the technology just gets you in the door. The show has just begun.

Most people Wolf writes about do it for personal reasons. One wanted to know how his coffee consumption helped him focus, another tried to cure his sleep apnea, and still another noticed that flax seed oil, or just lots of butter, improved his cognitive performance.

As in good BI, the experiments often raised new questions. And sometimes the new questions are unexpected, as in Wolf’s own experience.

Often, pioneering trackers struggle with feelings of being both aided and tormented by the very systems they have built. I know what this is like. I used to track my work hours, and it was a miserable process. With my spreadsheet, I inadvertently transformed myself into the mean-spirited, small-minded boss I imagined I was escaping through self- employment. Taking advantage of the explosion of self-tracking services available on the Web, I started analyzing my workday at a finer level. Every time I moved to a new activity — picked up the phone, opened a Web browser, answered e-mail — I made a couple of clicks with my mouse, which recorded the change. After a few weeks I looked at the data and marveled. My day was a patchwork of distraction, interspersed with valuable, but too rare, periods of focus. In total, the amount of uninterrupted close attention I was able to muster in a given workday was less than three hours. After I got over the humiliation, I came to see how valuable this knowledge was. The efficiency lesson was that I could gain significant benefit by extending my day at my desk by only a few minutes, as long as these minutes were well spent. But a greater lesson was that by tracking hours at my desk I was making an unnecessary concession to a worthless stereotype. Does anybody really believe that long hours at a desk are a vocational ideal? I got nothing from my tracking system until I used it as a source of critical perspective, not on my performance but on my assumptions about what was important to track.

I wish Drucker were around to respond. Wolf’s insight sounds like important stuff for everyday knowledge workers, especially those who work alone. What’s more important to a knowledge worker than time?

These experiments are often haphazard and highly personal.

Generally, when we try to change, we simply thrash about: we improvise, guess, forget our results or change the conditions without even noticing the results. Errors are possible in self-tracking and self-experiment, of course. It is easy to mistake a transient effect for a permanent one, or miss some hidden factor that is influencing your data and confounding your conclusions. But once you start gathering data, recording the dates, toggling the conditions back and forth while keeping careful records of the outcome, you gain a tremendous advantage over the normal human practice of making no valid effort whatsoever.

Yes, just as analytics gives companies a tremendous advantage over those who make less effort.

Let the BI traditionalists pooh-pooh self-tracking. The very same people might have dismissed such things as visual analysis, agile development, and at one time even business intelligence itself. Sometimes it take a few pioneers and geeks, perhaps even a secretary, to prove a concept.