Repeating the warning, though, makes you look like a party-pooper. In 2007 at the TDWI conference in Las Vegas, a keynote speaker raised it one morning. Jonathan Koomey — author of Turning Numbers into Knowledge and one of those voices the BI world needs more of — did his best. But I could see the unfolding disaster from my banquet table, as attendees glanced at each other in scorn. When the lights went up, not one person raised a hand with any question or comment.

Now Sex, Drugs, and Body Counts: The Politics of Numbers in Global Crime and Conflict, edited by Peter Andreas and Kelly M. Greenhill, tries it again.

You may wonder what sex, body counts, and politics have to do with data analysis, but try to keep an open mind here. The book promises to let us spit out the usual cud of business intelligence, data quality, and get to the real spice: the politics of data. I can’t wait to read it. For now, see Jack Shafer’s review on Slate.

I won’t be surprised if the book points out how each organization’s core group subtly chooses the stories its data tells. I’ve just finished Art Kleiner’s Who Really Matters, which goes into detail on these groups’ formation and influence, including how they define who’s in, who’s out, and why. It’s the essence of politics.

Though core-group members may not ever lay their smooth palms on any data, data is nonetheless coiffed to suit these people. Through layers of managerial interpretation and re-interpretation, their influence cascades all the way down to tiny decisions about how data’s summarized, what’s measured, how it’s measured, and who measures it.

Like other forms of expression within an organization — speech, email, jargon, attire, hair style, suit or T-shirt — data is part of the politics. Though this has a big effect on decision making, it seems rare that I find it on a BI-event agenda. BI’s scope needs to widen.

Sandy had been busy analyzing data. For years on Wall Street, he pored over mortgage-backed securities with a tool he and peers developed for themselves.

He only learned of ETL recently. He’d become acquainted with a data architect with whom he shared a bus ride every day to and from their offices in downtown Manhattan. “I had never really spoken to him before,” Sandy recalls. “He was in a different world even though we both dealt with data.”

Sandy described to him his rapidly maturing tool. As I imagine the scene, the calm data architect suddenly twisted himself on the cramped bus seat to face Sandy. “You don’t do ETL? You work with raw data??”

No, he didn’t do any ETL, Sandy explained. “We didn’t realize how important that was,” he recalled. “We had always just stuck the raw data into the database and then realized, ‘Hey, this data’s a mess.’” He instructed users to clean it themselves. “You get the data from the horse’s mouth. You’re the expert. We didn’t realize how powerful this was.”

In Sandy’s system, you don’t worry about database design. He and his partners not only didn’t worry about ETL, they wondered how data analysis could not be done their way — import first, clean later. “It makes good sense if you can get away with it.”

A crucial factor that lets the tool work as it does is speed. It allows the 1010Data engine to calculate and recalculate repeatedly. The summaries that cubes harbor for anticipated queries are no longer necessary. Parallel processing with a columnar database runs fast enough. In place of ETL, he uses what he now calls “ELTAR,” for extract, load, and transform as required.

A hurdle, he says, is conventional beliefs held by his sales prospects. In one phone call recently, he explained to a prospect that ETL was unnecessary. The man replied, “That’s not credible.” In fine sales form, Sandy said, “Then you’ll be impressed when I prove it to you.” The prospect replied more firmly, “You don’t understand. That’s not credible.”

Actually, the technology’s credibility doesn’t matter much. The company, 1010Data, offers reporting and analytics on the cloud — invisible to customers except for the results. Sandy says, “We could have monkeys writing on scratchpads.” To those willing to try, he offers to prove it with the prospect’s own data.

Their technology’s speed allows them to do the work of dozens with a team of a few people, he says, and to finish large data warehouse projects in weeks that would otherwise take months or years. If multiple customers use the same data, such as stock market data, the time required is even less.

All without ETL.

Consider this description by a BI analyst: “Tableau provides business analysts speed of thought visual analysis on data held in memory on their desktop machines.” All that’s fine, but it may as well have been about a whole bunch of other tools, too.

At the root of this fuzz, explained McDaniel, is that most analysts who concern themselves with tools don’t actually use the tools. They rely on demos , marketing, and hearsay.

Though much of McDaniel’s recent work has centered on Tableau — his second book is Rapid Graphs with Tableau Software and he gives training sessions around the country — he also has a long, credible trail back through BI and data mining. He was director of analytics at Netflix, and has worked with more than 50 companies in BI. His first book was SAS for Dummies.

“I love SAS,” he says. Still, he remembers his sister in law’s reaction to his book on SAS. She was not an analyst but a “people manager.” These are the ones, he says, who have hated BI because “it had been made into a priesthood.” When she had looked through the book, she said, “Oh, this is great” and put it down. But she read the Tableau book for a half hour and said, “You should come talk to some people I work with.” She had recognized what she could do with the tool.

McDaniel’s sister in law and many like her don’t care whether the data is “in memory,” they don’t see themselves as business analysts, they take “desktop” for granted, and they know “speed of thought” is just gloss.

The list of features really doesn’t matter. All that really matters is whether someone can do what needs to be done with the tool.

McDaniel imagines a throw down, a data analysis match. It would be open to any BI vendor. Each vendor would send their best people, and each team would receive a uniform set of data. Over some defined period, teams would analyze and then present the results to a panel of vendor-neutral judges.

The reward? Perhaps a signed copy of a Stephen McDaniel book, or maybe a beer, possibly both. But certainly, repute.

What do you think of the face-off idea? Please write a comment.

A book from 2000, The Social Life of Information by John Seely Brown and Paul Duguid, suggests that BI designers have neglected basic human needs. Jack Vinson, of Knowledge Jolt with Jack fame, has just posted a worthwhile review that sent me scurrying over to Amazon.

Failure begins early for many new, supposedly revolutionary information systems. Designers “assume that the way people operate with respect to information has to do with only the information. … But there is a social life that revolves around the information that is much harder to capture and codify,” Vinson writes. “We look to verbal and physical queues for validity of what someone is saying. Our business processes have much more than just the inputs and outputs.”

Jumping forward but on the same thread:

… in the essay on reengineering … the authors describe how all the social life around business process is downplayed and often treated as waste. Businesses were re-engineered to remove much of the social lubricant that helped business flow. The essay on knowledge management was hopeful that KM would be a shift away from the intense focus on information and account for the human aspects of knowledge: that knowledge requires a knower. They have a great phrasing: information can easily be written down and transferred. But it is much harder to detach (and transfer) knowledge from the know-er and the context in which that knowledge resides.

The book is still important even after 10 years. It doesn’t even mention business intelligence, yet it addresses some of its fundamental problems.

Take a look at The Social Life of Information on Google Books. I also recommend Knowledge Jolt with Jack. Always worthwhile.

Many of the objections were of the “if man were meant to fly, he’d have wings” variety. But many others were valid.

The practice will run over a few bumps before it joins mainstream performance management and business intelligence. Unlike the impersonal data we know and love, keeping data about oneself can be uncomfortable, difficult, and downright weird.

One of the articulate skeptics called it “robot envy.” In his weblog, Marginal Utility, Rob Horning summed up his objections in the final paragraph.

Numbers can provide only one sort of “truth” about ourselves, and to pursue it we must surrender or compromise other kinds of truth—for example, the intuitive faith we have in our qualitative assessments of our dasein. [...] In other words, we give up our soul for a spreadsheet.

I’d like to meet the spreadsheet that steals souls. Until then, I’ll cling to my belief that no spreadsheet, not even Excel, has any more power to do that than a blood pressure cuff or a bathroom scale.

A more credible response came on the New York Times site from “Matt” in California.

Self-tracking will undoubtedly be used to oppress. It will wend its way into mainstream culture, eventually becoming something that employers expect of you as a matter of course. The temporal “productivity gaps” which we use to daydream, think about politics or other non-work related ideas, or simply consolidate memories, will be targeted and eliminated. Also, it is almost inconceivable that self-tracking data will avoid eventually going public.

Wolf gave his own response to some of the criticism (apparently a few minutes before Matt gave his).

I think many of the critical reactions make sense. What are we doing to ourselves? But I suspect that even the people who say something like “turn off the computer and go outside” are more deeply involved in the culture of self-tracking than they realize, and would benefit from going beyond initial revulsion. We _are_ in the process of changing. Our new selves will have new capacities as well as new vulnerabilities. Literacy itself was once a threat to our humanity: it interfered with memory, and substituted external representation for interior experience. It replaced living dialog with marks on a page. But we found a new sort of humanity in this world of letters.

The easy answer is that self tracking has to be done in moderation. Assuming it catches on, we’ll see public-service posters on buses and trains warning against overtracking and out-of-control “self love.” But every good thing is overdone and always will be. — and the solution has never been to ban it, deny it, or belittle it. It’s here, it’s coming, and we might as well use it.

See the article here, the 59 reader-recommended responses here, and all 138 online responses here. See the 7 letters here.

As you may know, the KnowledgeSync tool monitors activity and generates alerts. Event A occurs and, bang, an alert flies off by email, text, perhaps even ICBM. New sales inquiry? The tool can even issue a reply.

What could SAC want to do with such automatic alerts? Don, who is VP of sales and marketing, recalls saying to his boss, “‘You can take this customer if you want to, but do you really want to take that support call?’” Nope.

He and I went on to fondly recall those two 1962 films, “Dr. Strangelove” and “Failsafe.”

Tracking your time then and now is personal, it’s messy, and it’s the essence of business intelligence: collecting data and reading it for guidance in business activities that matter. Is there anything that matters more to an organization than productivity of its people? For a small office or home-based business, this might be the best BI there is.

This gets no recognition in the BI industry that I can find, at least not in the conservative world of TDWI. At least not yet.

PI — for “private intelligence” — has different issues, starting with data collection. In BI, data comes from transactions, all recorded routinely. In PI, most of it has to come from a “secretary” or from our own, tedious notation.

I dabbled in it once. The insights were good, if painful, but mostly it was tedious. A few years ago, a confluence of personal events let me do something I’d always wanted to try: hole up for a few months in a Sicilian village I knew slightly. The food was good, I had relatives nearby, and the nearby church bells rang all day and all night, four times an hour. At the same time, I had a book to edit. To stay productive, I made a game out of the work, tracking my time to the minute in Filemaker.

I liked the local food and started to hate the book, an office manual that inadvertently revealed a con game. Even so, I threw myself at it every day. But no matter how hard I tried, no full day ever resulted in more than about two hours of actual, productive work. My “quick breaks” for walks and coffee with a friend actually took up more time.

I made a Filemaker database because I could find no off-the-shelf product that would do anything close. Each period, no matter how short, had a starting and ending times I entered with buttons, and a calculation field figured the duration. A drop-down menu offered my usual activites. I could make a report for any period.

I thought some product would do that better, but I could find nothing. Then the May 2 issue of the New York Times Magazine ran an article by Gary Wolf about this, “The Data Driven Life.” My Filemaker invention wasn’t too far from what others have used, and now new devices are coming along that could make all that seem so old hat. Some people are even sharing their data on the cloud.

But as in traditional BI, the technology just gets you in the door. The show has just begun.

Most people Wolf writes about do it for personal reasons. One wanted to know how his coffee consumption helped him focus, another tried to cure his sleep apnea, and still another noticed that flax seed oil, or just lots of butter, improved his cognitive performance.

As in good BI, the experiments often raised new questions. And sometimes the new questions are unexpected, as in Wolf’s own experience.

Often, pioneering trackers struggle with feelings of being both aided and tormented by the very systems they have built. I know what this is like. I used to track my work hours, and it was a miserable process. With my spreadsheet, I inadvertently transformed myself into the mean-spirited, small-minded boss I imagined I was escaping through self- employment. Taking advantage of the explosion of self-tracking services available on the Web, I started analyzing my workday at a finer level. Every time I moved to a new activity — picked up the phone, opened a Web browser, answered e-mail — I made a couple of clicks with my mouse, which recorded the change. After a few weeks I looked at the data and marveled. My day was a patchwork of distraction, interspersed with valuable, but too rare, periods of focus. In total, the amount of uninterrupted close attention I was able to muster in a given workday was less than three hours. After I got over the humiliation, I came to see how valuable this knowledge was. The efficiency lesson was that I could gain significant benefit by extending my day at my desk by only a few minutes, as long as these minutes were well spent. But a greater lesson was that by tracking hours at my desk I was making an unnecessary concession to a worthless stereotype. Does anybody really believe that long hours at a desk are a vocational ideal? I got nothing from my tracking system until I used it as a source of critical perspective, not on my performance but on my assumptions about what was important to track.

I wish Drucker were around to respond. Wolf’s insight sounds like important stuff for everyday knowledge workers, especially those who work alone. What’s more important to a knowledge worker than time?

These experiments are often haphazard and highly personal.

Generally, when we try to change, we simply thrash about: we improvise, guess, forget our results or change the conditions without even noticing the results. Errors are possible in self-tracking and self-experiment, of course. It is easy to mistake a transient effect for a permanent one, or miss some hidden factor that is influencing your data and confounding your conclusions. But once you start gathering data, recording the dates, toggling the conditions back and forth while keeping careful records of the outcome, you gain a tremendous advantage over the normal human practice of making no valid effort whatsoever.

Yes, just as analytics gives companies a tremendous advantage over those who make less effort.

Let the BI traditionalists pooh-pooh self-tracking. The very same people might have dismissed such things as visual analysis, agile development, and at one time even business intelligence itself. Sometimes it take a few pioneers and geeks, perhaps even a secretary, to prove a concept.

Some people associate that slogan with protests and even violence. But I think the best paths to power usually involve well-analyzed data, whether in public life or at work. Now the Little Guy has a potent new tool to deploy.

Lyzasoft founder Scott Davis calls Lyza on Lyza Commons “the YouTube of data.” This fully functional cloud-based version of Lyza is a strong tool for office-based, home-based, cubbyhole-based, dorm-based, or public library wifi-based users and groups. Import your data from whatever sources you have, refine it, share it with to whomever you like, and even charge toll over Paypal if you want to.

“Obviously,” Scott says, “what we’re doing is saying, ‘This thing can scale.’ But instead of going for the uber-enterprise as our leading play, we’re saying that what’s unique about this technology is it can make it to everybody within a small and medium business without having to have a big IT team around.”

Lyzasoft’s second, paid tier serves customers who need private clouds. That version starts at “small” for $150 a month, seating up to 10 users and providing “plenty” of storage. Go upward through “medium” and into “large,” and you pay $2500 a month for up to 200 users.

Wait, you say. You’ve heard this “YouTube of data” thing before. Yes, just three months ago another YouTube of data launched, Tableau Public. (I wrote about it here.) Tableau, Lyza, and YouTube itself all say “power to the people” by popularizing a medium with free, easy-to-use tools and a venue. Each one’s growing crowd of Little Guys and their audiences turns into a movement that the those in executive suites can’t help but notice. At some point, YouTube and those who follow its model hope that “free” leads enough customers to “ka-ching” to yield a profit.

YouTube seems to be well on the way. Its ready-to-roll movie theater had fired the imaginations in a waiting mob. These filmmakers-to-be had been trained over years of TV and movies to understand film and crave a chance to do their own.

Is there a waiting mob of would-be data analysts? One pioneer of free analytics is skeptical. LucidEra founder Ken Rudin, now vice president of analytics at Zynga, says you need more than free tools, no matter how easy the tools are to use. He says, “Tools are only as valuable as the questions you ask.” One of his biggest hurdles was getting customers to appreciate the possibilities of analytics.

But the YouTube idea is more than tools. It’s a game and a self-reinforcing mob. The tiny films YouTube users make don’t just play as if on a jukebox, they’re scored, they’re answered, and commented on. It’s like the difference between voting in a little booth and going out on a street march. It reinforces and stimulates. Unlike most business environments, it asks people to play, which is how Lyza Commons and Tableau Public users will break out into creative and incisive data analysis.

I also hope there’s a new supply of analysts. Ken Rudin and others are hungry for them. (In fact, if you’re a data analyst and you want to work with cutting-edge technology and data in one of the world’s largest databases, email Ken today at krudin@zynga.com.)

Power to the data analysts!

To keep her priorities straight, she keeps a “ruthless focus” on the library’s real needs. She keeps Maslow’s hierarchy of needs in mind, the theory that people satisfy needs in order, from basic needs like breathing all the way up to “self actualization.”

She tells about one upstate New York librarian she heard about back when libraries were first urged to go online.

A consultant went to visit a small library — one of those Barbie Dream libraries that are hot in the summer, cold in the winter, and staffed so minimally that the library worker covering the single desk will excuse herself to change the toilet paper and greet the UPS delivery person.

So the consultant explained to the library director that the online catalog could do this, and it could do that, and it would have all these marvelous functions, and the library would be so much farther ahead, etc. etc.

And the practical old librarian who had been quietly listening tilted her head and replied, “I’d still rather have a flush toilet.”

I hear so much about “solutions” that I think are probably not solutions at all. So I found K.G’s excellent post refreshing. See what you think.

When the lights went up, a “Bart” was right nearby me at the big round table. Though he “loved” Mark’s salute to his work, consultant Rick Paul wanted more.

As a Bart, he works with a lot of “Marges” and “Homers.” In Mark’s model, the Homers are the everyday business intelligence consumers, about 80 percent of most work groups. The Marges are about 18 percent, and they actually think a little. The last 2 percent are the Barts, the ones who analyze and invent — and who’re limited by the BI systems built for Marge and Homer.

The more painful obstacle facing many Barts, says Rick, isn’t about any technology.

He tells how his team started with three people, all data architects, all smart. “We could do anything,” he recalls. Now the team has 120 members, many of them Homers. They’re of the 80 percent who consume but don’t invent or even think very much. “They’ll fake inability,” he says, “to tempt or coerce the three innovators to do their work. They say, ‘Oh, you’re so good at this. It’ll just take you a few minutes to do this.’” It really does take only a few minutes, but “it’s not thinking work.”

“I’m lazy,” he says, “but when I don’t want to do something, I figure out how to automate it.”

Rick says he’s still trying to figure out how this situation can be resolved. He mentions isolation, but he also thinks of encouraging the 80 Percenters to have some vision for their own careers. They should have some way to “add intelligence to their own work on a daily basis. They should be actively engaged with their work.”

Smaller teams might also work, he says. In a team of 120, it’s easy enough to do nothing for weeks at a time. It’s much harder in a team of, say, seven members.

“The innovators have to be positioned to influence the company,” he says, “but not be abused.”