The Unknown
As we know, 
There are known knowns. 
There are things we know we know. 
We also know 
There are known unknowns. 
That is to say 
We know there are some things 
We do not know. 
But there are also unknown unknowns, 
The ones we don’t know 
We don’t know.

—Feb. 12, 2002, Donald Rumsfeld, Department of Defense news briefing
(From Slate) 

That Donald sure had a way with words, and I think they apply pretty well to our little world. In web analytics it is important to be keenly aware of what you don’t know. Why? Because we use the data we have – the things we know- to make decisions. Every day we’re trying to optimize the things we’re measuring, but often what we’re measuring is just a proxy for actual success. Because there are things we don’t know, things our tools can’t tell us. For exmple, if you run a lead gen site you probably don’t know if the leads are actually converting. If you’re looking at a ecommerce site what is the return rate on the sales you’re measuring? Are they repeat or one-time buyers? 

There are tons of critical connections that most of us haven’t made yet. And if we don’t take that uncertainty into account we can make some really bone-headed recommendations. We can go to great effort and expense to optimize the part of the process we can see and measure, while our seccess gets lost in the critically important part of the process we can’t see. Luckily for us, all is not lost. These are things we know we don’t know, so we just have to remember that, and factor it into the decisions we make from the things we do know. 

One thing I know is that I haven’t posted here in almost exactly a year. The year has seen me move from a medium sized retail company to a medium sized unit of a giant company. I’ve also come to realize why there are so few WA blogs from practitioners — we can’t really write about work. Consultants can write obliquely about what they’re thinking about all day, but that doesn’t pay for those of us working on a single site. That said, I’m going to try to make the effort. Will the next post come sooner than next April? I hope so, but that is a known unknown.

It was also very illuminating. I saw many things that clickpath analysis (even super detailed clickpath like Tealeaf) just wouldn’t and couldn’t show. And many things that really explain what it does show.

But what really struck me as odd was that after struggling with our site for an hour every person said “I like that site, it was pretty easy to use” or something similar. If those were good experiences, I shudder to think what a bad experience is like.

Tealeaf collects everything. I mean everything. Every cookie, every image request, every bit of information entered in forms, everything. Their bread and butter is allowing you to replay single sessions to see exactly what happened. It is an amazing tool in this regard. It has extremely powerful search capabilities that can sift through millions of requests and turn up the few that have the exact characteristics you’re looking for. Do you want to see all the sessions for people with customer numbers between 55550 and 55562? Tealeaf is your tool. Do you want to see what happened at 6:10 last night? No problem.

It also has very nice monitoring capabilities. You can fire an “event” if a particular sequence of actions take place on your site. We use it to monitor for a few rare but serious errors that crop up from time to time. Some use it to monitor for fraudulent activities or hacking attempts. The event can send an email or just flag the session so it is easy to find. I use events to find pages that do not have analytics tags on them. Tealeaf has fairly rudimentary reports that show how many and when events occur.

We also use it for ad hoc web analytics requests that would otherwise require a tag change. It’s not made for operational reporting, but if we want to figure out a particular question without going through the whole IT rigamarole Tealeaf is often ideally suited to do this.

Tealeaf also has a module called cxConnect which allows you to export all this information to a database. Every day or every hour or whatever it will spit it all out into a SQL Server database. At first glance this seems like a killer web analytics application  — just imagine, every little detail of all activity on your site all organized nicely for you in a database!

That is great until you give it about 30 seconds of thought. Sifting through this data is a herculean task. There is no way you have enough disk space to keep it all, so right away you need to write some sort of ETL to pick out what you need and compact the rest.  Then you need to replicate all the reports that a web analytics platform has out of the box. Will you end up with something better than you’d get from Omniture or WebTrends? Yes, you probably will, if you have smart data warehouse architects and good technical resources. If you don’t have both these you’ll end up with a large pile of useless crap. Will you end up spending way more money and time creating and maintaining this system than you would by buying and implementing Omniture or WebTrends? Absolutely, probably many times more.  Will it be worth it? Who knows. If it works it would be awesome. If it doesn’t… well, there are a lot of web analytics jobs out there, right?

The strength of the traditional web analytics applications is what they don’t show you. They sift out all the junk that you don’t need to see and they’ve created a bunch of reports that present the information that is useful. Someone else worries about the data storage/retention problem. It’s all built into your very predictable monthly fee. They may not be perfect, but they are pretty good, and buying one of these is much safer than embarking on a huge data warehousing project. As we all know there is a ton of detail in web data that is totally useless 99.9% of the time. Tealeaf lets you efficiently find that 0.1% – and that alone is more than enough to allow it to pay for itself.

What is going on here? Here are my votes:

• An interesting and cooperative strategy to get their links on each other’s branded search. It’s already worked for Avanash.
• Preparation for a CAGE MATCH at the next Emetrics.

Only time will tell, but I’m registering for Emetrics now just in case.

It now checks for MS Analytics and Google Analytics v2 code. For MS Analytics it outputs what would be sent if you called “TrackPage” on that page. I’m still not smart enough to decode the GA javascript enough to output what is sent, so it just lets you know that the code is present on the page. I also updated it to detect the latest version of WebTrends’ tag and I removed the VS check. I’ve learned a lot more about this product in the last year and I now know it was pretty ridiculous to make any sort of claim of reliably detecting it.

Due to WordPress’s very un-helpful smart quote feature I couldn’t put the bookmarklets in this post. You can find them in the right margin of my blog. Again, there are two versions of it. The first one works for IE7 and Firefox. For Firefox just drag the following link to your links toolbar, for IE7 right click and choose “add to favorites”. Then save it in the links folder.

This second one works for IE6. Right click and choose “Add to Favorites” and save it in your links folder. The downside to this one is that it will call back to blog.keyes.us each time you use it and it gives an error on https pages.
These updates will not overwrite the old version, so if they don’t work as well then a) let me know why and/or b) keep using the old version. These have been lightly tested so please let me know of any problems you encounter when using them.

Finally, if you’re interested in this you may like Stéphane Hamel’s WASP tool. It’s much more fully featured than this one, but it works differently. Also Eric Peterson has a hosted vendor discovery tool that you may find useful.
PS — somehow my comments table got corrupted, so all the comments on my previous posts were lost. Sorry! If your comment was deleted it’s not that I hate you, it’s that my Wordpress database hates you!

Today Ian Thomas points out Live 3d Stats which IMO suffers from a very bad name, but might actually be worth the money. One problem with working with web analytics is that it is really easy to lose context. At an eMetrics conference a few years ago someone suggested that you should “grep your log files” occasionally. By that they meant look at the log files as they are created in real time. This allows you to see visitors using your site in real time which really kind of brings to life the whole web analytics thing.

 Of course grepping log files isn’t for everyone, but Live 3d stats may be. I like the idea of having this displayed on a screen in a public place in the company just to bring home the idea that people are coming into the site from everywhere at all hours from all sorts of zany keywords. Of course, in reality the display would fade into the background after a few days. Sadly, no-one would “see” it after they got used to it….

Even so, realtime is cool in limited doses so this might be worth looking into.

 I also upgraded to IE7 and found that the self-contained firefox WA detector bookmarklet works fine. The IE one works as well, but the Firefox version is faster and doesn’t give an annoying security warning on SSL sites. If you’re using IE7 (or Firefox) you can use this version:

WA Detector (right click and choose “Add to Favorites” in IE, or drag to your Links toolbar in Firefox)

If you’re reading this via RSS you may have to click through to the site to get it to install correctly.

Enjoy!

 

It turned out to be pretty simple to implement and the result is quite useful and interesting. This code gives you a button on your browser; press the button and it will tell you which tags it finds on the current page — and which it doesn’t find. It currently looks for javascript tags from these vendors:

• Google Analytics
• Web Trends
• Omniture Site Catalyst
• HBX
• Coremetrics
• IBM Surf Aid
• Clicktracks
• Fireclick
• Visual Sciences

It works by checking for the existence of a common element in the vendor’s javascript. So, it can be fooled if the page happens to have that element in an unrelated script or if the implementation uses a non-standard javascript tag. Also, my definition of “common element” may very well be flawed for some of the vendors. I just looked at the code on a few sites for each one and it is very possible that that code was atypical. If you use it please let me know about any false positives or false negatives that you notice.

I also coded it to show what is being sent back to the mothership for WebTrends and Site Catalyst. If you really want this functionality for another tool let me know and I’ll try to work on it. Or better yet, add the functionality and send me the code! I’d really like debugging for Google Analytics so I can see exactly what is being sent back, but I can’t see an easy way to do it. Any help here would be greatly appreciated.

I think that’s all the provisos, addenda and quid pro quo. Here is the code (and of course there are browser compatibility issues):

• For Firefox just drag this link up to the “Links” toolbar on your browser (if you can’t see it go to View–>Toolbars and check it): WA Detector
• For Internet Explorer right click on this link and choose “Add to Favorites”. Put it in the links folder and it will show up on your links toolbar (if you don’t see it go to View–>Toolbars just like in Firefox): WA Detector

The browser problem arises because IE 6 will only accept bookmarks that are 508 characters long, so the IE bookmarklet uses an external script file that is hosted at blog.keyes.us. Feel free to use it as is or to copy the file to your own server. I make no promises as to the reliability or longevity of this site.

Also, if you prefer, I made a Pirate WA Detector. It uses a external script file and it should work in Firefox or IE.

Enjoy, and let me know what you think!

Update: I should have mentioned this above, but there are two other vendor discovery tools that I am aware of. Eric Peterson hosts one on webanalyticsdemystified.com and Webbanalys has a desktop application. It also looks like Stephane Hamel is working on one. Stephane also has a nice roundup of the tools here.