This sounds like the answer to our engagement prayers: one metric to measure all.  It’s the Esperanto of engagement, a common language by which we can understand the customer.

Unfortunately, I think it’s a terrible idea. In many ways, it’s the antithesis of all that measurement stands for in my mind.  Why?  Let me explain.

The Basic Premise of Measurement
The basic premise of measurement is that you want to measure something so you can improve it, if necessary. If my body/fat ratio is out of whack, I’ll work out and eat better to bring it back in line. If my conversion rate is lower than my historical average, I’ll try to improve it.  If campaign response is weak, I’ll look at some fresh creative.

It’s pretty simple – collect data, analyze, improve.  I love this because of its simplicity and objectivity. In my early days of analytics, I spent countless hours watching as executives argued emotions instead of facts.  And, unfortunately, back in the late 90s, analytics were hardly robust enough to confidently argue in favor of either side.  Generally the person with the bigger title won the argument and their recommendations were put into place.

But now, analytics are far more robust (when implemented and managed properly), and we live in a wonderful world of objectivity (for the most part).

So what happens when you start combining metrics into uber-formulas like Visitor Engagement?  That model breaks, because you introduce a level of abstraction on the data. You “dumb it down,” introducing bias and subjectivity.

Breaking the model: why uber metrics don’t work
Let’s say ‘engagement’ is classically defined as leads/visits on the site.  That’s an objective measure of how a visitor’s experience is leading to a positive outcome for both parties.  In other words, it’s a measure of how engaged the visitor has become in his relationship with a company, and it demonstrates a strengthening relationship – all good things in the world of customer management.

Now let’s say you create an engagement mashup.  The mashup includes visitors that have returned “often” to the site as one metric, when they view “important” content as another metric, and, just for good, measure, we’ll include visitors that spent a “long” time on the site as the final metric.

That’s just three metrics; it can’t be that biased, right?  You bet it can.

First, what kind of return frequency is “often” - two visits?  Four?  Six?  That’s subjective.  What is “important” content?  The home page?  An article?  A support document? Subjective again.  And what is a “long” time on site — 5 minutes, 10 minutes?  Perhaps “long” means any visit that exceeds the average for the site that week?

You can see how quickly this becomes totally subjective.  Because of its subjectivity, it has become totally worthless.  You have introduced massive bias without coming up with a metric that allows you to make decisions.

Let’s say this formula yields a Visitor Engagement “Score” of 40 for last month.  This month, the same formula produces a score of 30.  That’s a pretty dire situation — but what do you do about it?  How can your executive team act on that number?  They can’t!  Your best hope is to begin dissecting the Visitor Engagement score to its fundamental metrics and figure out which one is responsible for the decrease.

For example, suppose return frequency was flat, visits to important content skyrocketed, but time on site fell through the floor.  You’ll probably want to focus on time spent on site, and see if you can improve that.  But if your primary KPI of leads/visits has increased (i.e. your conversion), maybe you’ve actually done a really good thing and you should leave it alone. You’ve created a more frictionless experience, and the declining Visitor Engagement score supports this.

At this point, you’ll face the undesirable task of convincing your execs that the Visitor Engagement metric, which you fought so hard to socialize and adopt, should actually decline.

But WAIT! Not all uber metrics are bad
So I think you get the point.  Visitor engagement formulas are largely another fad, just like parachute pants and the Hollywood diet.  It’s a measure some consultants and vendors can pitch like snake oil.

But, that is not to say that uber metrics are completely worthless.  In select cases, you can actually leverage uber formulas to make very useful decisions.

Uber metrics that are purely objective can hold value to an organization.  Perhaps one of the greatest is RFM – Recency Frequency Monetary.  In that case, you’re dealing with an (almost) entirely objective uber metric.  For those not as familiar with RFM, it’s a classic customer segmentation technique that essentially calls for you to score your customers based on their ‘relative’ rank to one another along three primary metrics.

You then roll up these scores to arrive at an uber score, and identify your best (highest scoring) and worst (lowest scoring) customers.  Action you can take from learnings gleaned from this analysis are too numerous to name.  It’s actually a lot of fun to do these kinds of models. But even in this case, subjectively can often enter the picture.

For example, the timeline over which you analyze customer data is one of the principal points of subjectivity.  In the RFM model above, do you analyze behavior over 1 month, 6 months, 1 year or 6 years?   Maybe you just take as much data as you can find and mix it all together and hope for the best.  In turn, once you complete your RFM segmentation, what time period do you compare it to?  Weeks?  Months? Years?  Again, subjective.

RFM has a long history of being valuable – so again, I’m not throwing uber metrics under the bus entirely.  Still, I wouldn’t waste your time with most of them.  There are so many opportunities for optimization based on primary key performance indicators like conversion that you can keep your entire team busy for years.

Don’t try to build a better mouse trap, when you’re not taking advantage of the one you’ve got today.

So, those are my thoughts.  As always, I welcome your ideas and feedback.

As with any relationship, enthusiasm for web analytics doesn’t grow on its own, and it can’t live in a vacuum.

Often, what I see happening is that marketers begin generating reports and then don’t do anything with them. It’s like they’re expecting that one day, the report will jump up and do something, without any marketer involvement. When the needles don’t move on the reports, the love begins to die. But it’s not the fault of the analytics.

So consider these few simple ways to rekindle the love:

Step # 1. Remember what started the romance

When you implemented a web analytics program, you likely considered your key business requirements and looked at the business questions you were trying to answer.

Take a look at those questions again. What is the data telling you about those questions? Come up with a hypothesis, then run a test.

For example, was one of your goals to improve conversions? You might hypothesize that you have too much content on your home page. Maybe reducing the blocks of content to bullet points will help improve conversions.

Step # 2. Quick wins

People inherently gravitate toward success. After looking at your key business questions and coming up with a few hypotheses to test, pinpoint one or two of those things that are easy to implement and that might help you get a quick win.

Make it simple. Figure out what you may be able to do better, test it, and then measure it. Once you have a good success story, people become keenly interested.

Suddenly, instead of trying to force analytics down their throat, they’ll be coming to you.
The relationship will blossom.

If you don’t recall the business questions you were trying to answer when you implemented a web analytics program — or missed that step altogether during the implementation — give us a call. We can help you outline those questions, based on both your industry in general and your company specifically. We can also help you understand how to look at the data and come up with actionable steps to answer those questions.

Now I come to impressions, a big ugly elephant in the room that’s been lingering around for years…

When advertisers buy ad space, they actually buy “impressions” more than anything else.  You are likely aware that this is called a CPM model - or cost per thousand impressions purchased. A $20 CPM rate simply means it costs you $20 to buy 1,000 impressions on a site.

(Sure, there are other models like CPC - cost per click - which is the standard in paid search.  And there’s CPA - cost per acquisition - which resurges every now and again as an alternative but has yet to become mainstream for a variety of issues.)

In any case, if you look at firms that use audience measurement services to determine which sites they want to advertise on, it’s almost always based on CPM. This means that while unique visitors, time-spent-on-site, and page views are all well and good, in and of themselves, they are secondary metrics when it comes to the true nuts and bolts of ad buying.

So ironically, despite all this media fanfare about time-spent-on-site and the death of the page view, there is still a great divide between what advertisers and publishers use to evaluate advertising opportunities and what they use to actually execute on those.

Audience measurement firms are publishing secondary metrics that may help guide advertisers to a particular site, but they are generally disconnected from the advertising inventory itself.

Strangely enough, advertisers tend to purchase inventory based on a metric - impressions - that is rarely made available to anyone outside the company.

To be fair, because advertisers often buy only a portion of available impressions, you could argue that it doesn’t really matter if they know how much total inventory a site can offer. For example, if a site has 10 million available impressions, but you’re only in the market for 1 million, what does it matter if they have 1 million available or 10 million?

I believe it matters for several reasons.  First, saturation - if they have 1 million impressions and you want 1 million - that’s 100% of inventory. If they have 10 million, your 1 million only represents 10%. Depending on your ad objectives, 100% saturation or 10% saturation may be more beneficial.

But without the total inventory number, you have no idea.

Second, ad impressions can vary greatly by site.  This is because sites offer a variety of ad placements for sale - often more than a dozen. From simple home page banners to run-of-site ads to streaming media pre-rolls, there is a myriad of placements advertisers can offer - and often, more than one advertiser shares a particular placement, rotating depending on how many impressions have been served and how many have been purchased.

Fortunately the IAB has published - and continually updates - ad placement standards, so there is a general consensus as to different placement types.  But there are no real guidelines on how many or where these placements appear. For example, one site could have a single banner available on its homepage, while another could offer 10 placements, including one for streaming media.

Because unique visitors, page views, and even time-spent-on-site have a one-to-many relationship with ad impressions, it’s impossible to use those base metrics to even estimate impressions.

So take a step back, and a deep breath. If you’re with me so far, you’re probably asking why audience measurement firms don’t just publish impressions in addition to the other metrics.

Good question. One challenge is that ad impressions are difficult to measure for audience measurement companies.  An impression can be generated in multiple ways; there are few “ad serving” standards.  And publishers frequently combine several different technologies to serve ads, often on the same page!

Even if ad impression serving technologies were standardized, and the audience measurement companies could track them, there is another challenge (It’s something I talked about before, in the second post of this series): Panel-based methodologies are just not as good for granular reporting as are other methodologies like web analytics.  And many ad campaigns are just that - granular, targeted initiatives that are muted across user panels, if not altogether absent.

Furthermore, ads often change by day - whereas most audience measurement companies report weekly or monthly at best.

So don’t hold your breath on this one. I just don’t see it in the cards.

What should advertisers - and publishers - do?

I think there is a significant opportunity for advertisers and publishers to create a more efficient marketplace.  In turn, these efficiency gains should drive a better customer experience - as visitors enjoy a more relevant online (and offline) relationship with vendors.

How?  Take back control and set a new standard that is publisher-centric.  Let me explain.

First, imagine a world where publishers used web analytics to capture ad impressions.  Since these impressions are measured at the browser level, and do not even rely on cookies, they are arguably the most accurate measure you can achieve for true impressions.

Second, imagine a world where publishers also captured clicks, and even “downstream” success events (conversions, purchases, form completions, etc.) that occurred as a result of those clickthroughs.

Finally, imagine a world where publishers then provided this information to prospective advertisers, who in turn could evaluate if and to what extent they wanted to purchase that inventory.

A fairy tale, you say?  Nope.

The reality is that publishers can do this today, and many are already doing some of it using web analytics.

At Omniture, we have already helped customers capture impressions, in real-time, for their website and report those back to prospective advertisers.

This approach also captures any clicks that result from these impressions, and reports them back in real-time.

Of critical importance, it is possible to segment this impression-tracking with profile information that customers voluntarily and willingly provide you, all accomplished in accordance with your privacy policy.

For example, you could see the geographic distribution of all people that clicked on one of your home page banners. Alternatively, you could report on advertising and content affinities that would actually help you sell even more inventory on your site.

Taking this concept even further, you could segment ads, by demographic information that your customer’s have shared with you (and you have once again covered in your privacy policy). This would allow you to share gender, age, and income preferences with prospective advertisers by each ad run on your site. 

And taking it even further, you could use third-party data sources like PRIZM to augment the data even more – which I can imagine would provide some compelling multi-channel synergies given PRIZM’s prevalence in the offline world.

From this point, the key is to establish a vehicle or marketplace by which you can share this information with prospective advertisers.  This could be done as a collaborative exchange between numerous publishers.

So forget about taking back control from the audience measurement firms- you’re completely changing the game with this kind of data!  Existing approaches can’t even scratch the surface when it comes to providing this kind of insight.  The depth, the speed, and the accuracy are unparalleled.

Perhaps you think this is impossible; there’s just too much weight around the current approach.  Try telling that to Google, Yahoo, Microsoft or any other paid search engine.  They run their entire business on a model that’s not too dissimilar from what I’m advocating.

Think about it. Why does this work only for CPC and search keywords?  While there are some nuances to publishing that need to be considered (as I’ve highlighted above), it’s not radically different.  And paid search is a $7 billion market - so the model has legs.

Is this approach without its challenges? Of course not.  Standards will have to emerge, driven by governance bodies like the IAB.  And that will happen over time, just as it has for search.

But it won’t happen at all without publishers demanding a better mouse trap… a Brave New World, if you will.

So, long live the impression! And if you have questions about measuring and optimizing impressions, I encourage you to reach out to us.  We’d be happy to help.

I believe it is actually impressions that should become the standard by which both buyers and sellers of online media will begin to negotiate their buys. And perhaps, in the future, another metric like clicks will gain more footing in the non-search advertising world (clicks are already the standard in search advertising).

For a quick summary of my argument, read the first in the series. For more on time-spent-on-site, read on…

With its announcement, audience measurement firms have suggested that time-spent-on-site is a more equitable way than page views to measure visitor engagement in a Web 2.0 world.

The rationale is that Web 2.0 technologies like AJAX do not adhere to the traditional page metaphor, so time spent on site is a better form of engagement measurement. (For a description of why pages that use Web 2.0 technologies are difficult to measure using traditional page views, see Measuring Visitor Engagement: Brave New World or The Emperor’s New Clothes.

To that end, Scott Ross, director of product marketing at Nielsen commented, in a Beet.tv interview that, based on everything that’s going on with the influx of AJAX and streaming, total minutes is the best gauge for site traffic.

By way of example, Ross referenced, in the same interview (or read about it in a ComputerWorld article from Oct. 7, 2007), a comparison between MySpace and Youtube, where MySpace outnumbers YouTube’s page views by a factor of 10 or 11, but the ratio of time spent on MySpace is about 70% less - at roughly 3 to 1.

Sites like AOL and Yahoo also catapult to the top of the “engagement” list, driven by interactive applications like Instant Messenger and email.

With all of this in mind, it becomes increasingly evident why time spent on site has attracted so much attention. In a way, audience measurement firms didn’t have a choice. They needed to move beyond the page view to measure engagement, or the “quality” of the visit (but not necessarily the quality of the visitor).

But something else happens with time-spent-on-site. Audience measurement firms regain the high ground. Why? Because time spent on site is actually dictated by the panelist, not by the site they visit. In other words, time spent on site is measured by a meter that sits on the user’s computer - and unlike page views, sites have little way to manipulate or corrupt the number.

At this point, you could argue that this is great - kill two birds with one stone. Improve engagement or quality-of-visit measurement, and offer a metric that can’t be dramatically biased by sites themselves.

While I haven’t seen any articles mention this angle of the business (control vs. not), it’s nonetheless supportive of the claims that this is Brave New World territory and better for the industry as a whole.

But as I mentioned earlier, I actually disagree.

Time-spent-on-site has been available for years as a standard website metric.  In fact, it has changed very little since the prehistoric era when server log files were used to measure site traffic (yes, it was available back then).

So is it really a step forward for audience measurement - an industry that has recently come under fire for major challenges in their underlying methodologies and data accuracy?  And is it really a step forward for advertisers and publishers - who have likewise struggled to create an effective marketplace for buyers and sellers of ad inventory (hence the meteoric rise of third-party ad networks)?

I think not.  Next time, I’ll write about impressions, and why using impressions offers a significant opportunity for advertisers and publishers to create a more efficient marketplace.

As with other advertising mediums, the online audience measurement industry was born out of the need to provide publishers with a common currency by which they could market and qualify their sites to prospective advertisers.

If I’m an advertiser looking to reach 1 million people with a new movie promotion, how would I know which media sites I should advertise on? If I had a common metric - or a common set of metrics - I could quickly scan that list to find sites that reach 1 million people, and then buy that space to reach my audience.

For such simplified audience measurement like this, Nielsen is the de facto standard in the offline world, and at the outset of the Internet, there was no online equivalent.

In the late 1990’s, audience measurement firms sought to be this common currency, offering up metrics like unique visitors, page views, and time-spent-on-site. Nielsen was one of the first to throw its hat in the ring. They created a service that projected these metrics for major Internet sites, based on a panel they maintained of several thousands users.

This is nearly identical to their offline approach, and why not: if it worked in the offline world, why not give it a shot online? The challenge is that this panel-based approach is easily skewed and only useful at a very high-level. As I’ve talked about in the past, that’s because the Internet offers the potential to successfully reach people in extremely narrow niches of interest. If you’re a knitter who also likes to quilt but who hates to crochet, there’s probably a website for you and others with the same likes and dislikes. On the other hand, it is highly unlikely that, even with its panel of thousands, an audience measurement panel will have many knitting, quilting, crochet-haters on its panel.

Audience measurement firms will likely then struggle to measure the niche-y craft site, when in reality, that site may see tens of thousands of visitors per month. A yarn company looking for places to advertise, but who goes only by panel responses, may miss out on the site completely, never knowing there was a small but important group of crafts enthusiasts potentially eager to see the yarn company’s ads. And as many folks know, loyal customers can be 7x more valuable than new customers, so tapping into this niche customer segment is critical.

Along these same lines, targeted direct marketing initiatives like email campaigns, paid search, new microsites, etc., can also be understated by such panel services. Again, those initiatives are likely to hit only a handful of the panelists, and a “handful” is generally viewed as not being statistically significant enough to surface as a meaningful trend or change.

Similarly, when sites add new content - new articles, special editions, etc. - these can be understated or undetected. By how much? There’s no way to tell for sure, unless you use web analytics, which is arguably the most accurate way to measure the success of these initiatives.

Still in doubt? Run a simple test. If you’re a retailer, look at how many orders you have on a given day as reported by your commerce engine. Now check your web analytics platform. The orders, generally speaking, should be within 2-3% - if not perfectly in line. Now, check with an audience measurement firm – what are they reporting for the day? I’ve done this multiple times and never seen anything close to accurate. If you’re not a retailer, pick something else – like leads, applications, etc – that you can validate not only with web analytics but a back-end system. The key to this exercise is triangulation so you need at least one more data source beyond your web analytics and audience measurement services.

Of course, site-side analytics has historically offered very little to advertisers in evaluating competing sites, so I readily acknowledge that audience measurement can be a useful proxy for comparative traffic levels (as I’ve written about in the past.)

Still, in the late 1990s, when audience measurement firms introduced these panels, advertisers were understandably excited, because at least they could compare one site to another with the same metrics. In fact, for some time, venture capitalists and investment bankers often used these same services to estimate valuations for pre-IPO internet companies, using unique visitors as the measure of “eyeballs” the site could presumably monetize into paying customers some day.

Around the same time, page views also came to be viewed as a measure of engagement. Folks began to realize that not all unique visitors are created equal: two sites that each have 1 million visitors can be very different from each other in terms of reach, if most of the visitors to one of the sites come to the home page and then leave immediately, while visitors to the other site stay and browse.

Page views, then, became the check and balance against unique visitors, and ideally the two taken together could provide a rounded assessment of site engagement and revenue potential.

The challenge with page views is that they are actually not standardized. Nielsen and other audience measurement firms could control unique visitors because they managed the panels themselves. They paid or otherwise compensated each member of the panel so that uniqueness was fairly well preserved.

But audience measurement firms do not and cannot control page views because they source from the sites themselves. Pages come in all different shapes and sizes, some with dynamic content and some that are completely static. Not all page views are created equal – and audience measurement firms are faced with the impossible task of trying to create a common standard. And let’s pretend for a second that this was achievable…that audience measurement firms had picked apart every web page from every site, and classified it as a page view. Well, Web content can change multiple times per day per site so while the utopian standard could have theoretically been accurate, it would have quickly become inaccurate as content and layout changed.

For example, there is the standard HTML page that we all know. That’s fairly easy to standardize across sites.

But then there are generated pages with dynamic URLs such as retail websites that create new URLs on the fly for each product. What do you do with that? On top of that, you have dynamic pages that do not change the URL at all (see my example about the GAP in my previous post.

And streaming media and widgets are not even pages - they are complete experiences in and of themselves.

As the Web has evolved, these “non-traditional” pages have become increasingly prevalent, because in many cases, they provide a superior customer experience.

So while page views emerged as an early measure of engagement, it really was never fair to compare it across sites - whether they were tracked by a panel or otherwise.

In my next post, I’ll discuss audience measurement firms’ “new” metric, time-spent-on-site.

The credibility of audience measurement data from firms like comScore, Hitwise and Nielsen has been plagued for years, for a variety of reasons, including fundamental limitations with their data collection methodologies, unpredictable revisions to those methodologies and results from different firms that contradict each other.

Recently, some firms have been claiming that they have hit on the best and most effective metric for online advertising: time spent on site.

And I believe they’re dead wrong.

I believe the best way to measure online advertising is by the very metric we most often use to sell it: by impression. And, while there are a few potential problems in using this as a metric, the problems are not insurmountable.

An “impression” is counted any time an ad is “served” to an end-user, and it is the metric advertisers most often use when they purchase ad space (the commonly used “CPM” means cost per thousand impressions). Yet the metrics that measurement data firms have used in the past have little to do with impressions. Rather, measurement firms and the online advertising world have experimented with a number of very different measurement options.

Beginning in the late 1990s, we focused on unique visitors – largely because this was believed to be unique people. As I’ve discussed in the past, unique visitors are anything but unique people and this metric is fundamentally flawed. As we grew more sophisticated, we began to measure how visitors interacted with content on the site. Page views emerged as the de facto standard for measuring content consumption, and for a long time, that metric reigned supreme.

But now Web 2.0 technologies are completely disrupting the status quo and the page view is no longer as accurate as it once was. With Flash, AJAX and streaming media technologies becoming more prevalent, the page view increasingly fails to capture the complete user experience. That’s because these technologies deliver a richer, more dynamic experience that does not require a new page view with each new piece of content.

The Gap.com website provides a great example. If you navigate to the Jeans category, you’ll quickly notice a Flash-based module in the center of the page that shows different jeans on actual people. It’s pretty slick and you’ll notice as you click on the different types of jeans, a new person will walk out with those jeans on. But you’ll notice something else. Each time you click, the full page does not change. It remains the same, even though the user is able to interact dynamically with fresh content.

Thus, whereas in the past a visitor would have gone to a second - or third, or fifth - page in order to get the information about each subsequent style of jean, she can now remain on a single page. So in this example, is the single “page view” that the visitor saw really the best measurement of that visit, when she actually browsed through several products? Definitely not.

With this in mind, it makes sense that audience measurement firms are exploring new ways to quantify visitor engagement on the web.

What makes less sense is that they have chosen time-spent-on-site as the best measure of audience engagement. The move has attracted a good deal of press, some of which has gone so far as to suggest this is a ground-breaking step.

In coming blog posts, I will go into detail about page views, time spent on site and impressions. I’ll explain why time spent on site is not, contrary to what much of the trade press claims, the best available measurement. I will show that, while those metrics are interesting proxies of ad inventory, they are secondary metrics when it comes to the true nuts and bolts of ad buying, and they do little or nothing to help advertisers and media publishers connect with their target audience.

And I’ll prove that measurement by impression - the same metric that the majority of web publishers use to sell online ads - is not only desirable, but is eminently attainable.

To me, this decision to measure time spent on site is a case of the Emperor’s new clothes - the Emperor, and nearly everybody else, claims his clothes are the best ever, when in reality, he’s wearing nothing more than air.

It is rare in this world that we get to experience public vindication of our ideas. So it was with excitement that I read on the SeekingAlpha blog about Yahoo president Susan Decker’s comments during the company’s fourth quarter earnings call. I must have missed this when it first came out with all the post-Christmas excitement, but I revisited the earnings release just this past week in light of all the Microsoft/Yahoo chatter and what a gem I found!

Decker said that, moving forward, Yahoo would use visits rather than unique visitors as the most relevant metric for tracking the relative success of Yahoo sites.

She said:

“With consumers accessing the web in so many ways, we’ve looked for a more unifying global metric that’s more flexible across Yahoo’s and our partners’ properties and useful across multiple devices and geographies. We expect to use visits to Yahoo’s global starting points and anchor sites to be the most relevant metric going forward.”

She points out that the metrics that have been discussed in the past, such as uniques and page views, “may not tell the story of what’s happening and the key, value-creating starting points for consumers and advertisers.”

Ha! Let me bask in the light of a swift moment of “I told you so.”

Almost exactly two years ago, I wrote a blog post in which I said that, as a metric, visits were far more useful than unique visitors when tracking web performance. And, though I received plenty of kudos from others within the industry, it must be said that I got my share of “what could this wacky guy be thinking?” type of responses. The guys at Future Now, for example, were particularly scathing in their grokdotcom blog.

My theory was based on a few simple (or, as grokdotcom called them, “simplistic”) reasons:

1. Visits are more accurate than unique visitors.

2. Every visit represents an opportunity to persuade or convert a visitor to a customer.

3. Measuring visits is based on fairly established industry standards.

I explained each of those reasons in detail. I also continued to explore the reasoning behind my theory in other posts, including one where I laid out 15 reasons why all unique visitors are not created equal.

As I pointed out then, users access the Internet via a variety of browsers and a variety of computers.  Also, multiple users can access the internet via a variety of browsers on a single computer.  Users delete or accept cookies on various browsers and various computers. At any given point in time, then, these scenarios are being played out by your visitors to your website.  They are inherent in unique visitor counts and, by their very definition, make unique visitor counts completely unreliable.

Today, the issue gets even more complicated.  Users access the Internet not only by a variety of different computers (their home computer, their computer at work, their husband’s or wife’s computer) but by a variety of different devices including Blackberries and iPhones.

Times are changing, and the world of web analytics must change, too.  If you didn’t take my word for it, back in 2006, that visits was a more accurate measure than unique visitors, think about what Susan Decker of Yahoo said.  And Susan wasn’t alone either – several other large media concerns have explicitly or implicitly gone this same direction.  Then, go back and read or re-read my 15 reasons unique visitors are not created equal.

Consider that this might be just the first indication of a sea-change that is taking place in the world of web analytics. New devices, new technologies and new ways of using the web continue to create the need for constant vigilance on the part of web analytics companies.  We can’t afford to sit back and rely on what has worked in the past.

That’s part of the reason I was so pumped when I read about Yahoo’s shift.  It means we at Omniture are moving in the right direction.  We’re successfully staying on top of the changes the web world is encountering.

I’ll continue to explore these changes, and to share my thoughts with you.  Hopefully, they’ll help you to more easily navigate the choppy but exciting waters of a Web 2.0 world.  In the meantime, let me know your thoughts, even if you disagree.  As always, I look forward to reading your comments.

]]> http://blogs.omniture.com/2008/05/19/at-last-vindication-on-visits-vs-unique-visitors/feed/ http://blogs.omniture.com/2008/05/19/at-last-vindication-on-visits-vs-unique-visitors/ http://feedproxy.google.com/~r/omniture/blogs/author/Matt/~3/Zrx1G345hYE/ http://blogs.omniture.com/2008/04/21/analytics-goes-global-and-guess-what-it-is-easy/#comments Mon, 21 Apr 2008 16:17:04 +0000 Matt Belkin http://blogs.omniture.com/2008/04/21/analytics-goes-global-and-guess-what-it-is-easy/ Greetings everyone! It’s official – my office has become an aluminum tube at 37,000 feet as I travel the global presenting at our international Omniture Summits. Since our Salt Lake City Summit in early March, I’ve been down to Sydney, over to Paris, and just this past week up to Copenhagen. This Tuesday, April 22^nd is our London Summit, followed by Munich on Tuesday, April 29^th. It sounds exhausting, but it’s actually been a wonderful experience meeting with hundreds of analytics people all over the world. In this blog post, I wanted to share some of my observations from the road, and also provide some additional perspective on why analytics really is easy.

First off, wow! We had over 2,200 attendees at the Salt Lake City Summit, and nearly that many across our Summits in Australia and Europe so far. It’s unlike anything I’ve seen in the analytics world. In Salt Lake, we had companies with 5, 10 – even 20 people attending the Summit. All of these folks with a stake in analytics and optimizing their business. It was incredible.

Internationally it’s been a bit different. I haven’t seen the large teams that I saw in the US, but I see one-person armies with considerably more responsibility and drive than their US counterparts. In some respects, it is almost like the international folks are trying to do as much with one person as the US teams are doing with many. Because of the practical limitations of one person, I’ve noticed many international customers are focusing on 3-5 major areas they believe can make a massive difference, rather than 20-40 different analytics initiatives that can sit on a US customer’s agenda. Of course, I’m generalizing, but it’s been fairly consistent across the difference cities so I think it’s a warranted observation.

Another observation is the interest in automated optimization, namely Omniture Test and Target. I can’t even recall how many conversations I’ve had about the platform and the business value it can bring to an organization. Perhaps the interest has been so strong because of my first point – customers see not only the ROI potential of testing and targeting, but equally that they can automate it and leverage their already scarce resources. In any case, the interest has been palatable and I’m looking forward to talking more about it in London and Munich.

Customers are also looking to plug more “stuff” into their analytics platform. This can be anything from partner integrations like email, search, and sales force automation to data warehouses and off-line transactions. I’m really quite surprised how innovative some of these initiatives are, and how for the most part, people have retained their focus on the underlying business question. The good news is that our Genesis network continues to expand to support many of these partners and I’m looking forward to the fruits of these initiatives in the coming weeks and months.

Lastly, analytics is easy. This point has been underscored in countless conversations I’ve had on the road. I know there’s a blogger floating out there in the industry that likes to argue this point, and that’s perfectly OK with me. He’s trying to make a living based on the notion of analytics being complex so what else would you expect? Ironically, Omniture Consulting represents one of the largest strategic consultancies for analytics in the world - and if anything, you’d think I would want to say analytics is hard so we could likewise try to benefit from this perceived complexity.

But I don’t. Because it’s not. In fact, our team is successful because we deliver measurable value to customers. That’s it. While on the road, I’ve met with customers who’ve identified ROI opportunities in 30 seconds from analytics. I’ve also met with customers that want to learn how to achieve similar results. Not because they are complex – but simply because they don’t know how to do it. They simply haven’t been taught. And that’s a true gap in the industry that I have seen everywhere I go. Comments throughout the web, even in response to this blogger, underscore the point. People want to know how to be successful…they want to learn to drive…does it mean driving is complex? I think not. It just needs to be taught. It’s shortsighted to say analytics is hard just because some people don’t know how to do it. There is an education gap, not a complexity gap. If I can sit with a customer and show them how to derive millions of dollars in value in 2-4 minutes, teaching them to be self-sufficient and repeat this value-based optimization on their own, how can that be complex?

To be fair, I completely understand it’s not always easy to get people to change their behavior in response to analytics. In other words, it can take weeks, months, or even years for a company to change its behavior based on an analytical insight. I fully appreciate that, I’ve lived through it and even left a company for this very reason. But I’ve also worked at a company that would change the homepage within hours of observing a critical opportunity for improvement. Does that mean analytics are hard? Absolutely not. It means changing human behavior and perception can be hard, and that’s build into our DNA…if you’re fortunate, you’ll find yourself at an organization that is adept at change and improvement, or even a culture like the Japanese that are maniacally focused on improvement. Or you can find yourself in a situation where people are massively fearful of change, and you need to figure how you can build credibility and affect change in this environment. Those are challenges, yes, but they do not mean that analytics is hard.

So get out there, take action on your data, and improve your business with analytics. It’s easy. And if you think it’s hard, send me an email, call me on the phone – whatever you prefer – and we can train you and show you how easy it can be.

When deploying analytics, you should aim to reach as broad as possible across all your customer touchpoints. You need to if you want to be considered strategic. Your ability to optimize and understand the customer lifecycle is directly related to how comprehensive you can be with measurement.However, just because you go broad does not mean you should go deep. It is of much greater value to understand all your touchpoints at a minimum level that to understand a few touchpoints at a maximum level. I’m sure some people will disagree with this point vehemently – and I welcome the feedback. I’ve arrived at this conclusion based on my own experience and this has worked for me and the clients I’ve worked with.

In the world of analytics, it’s better to go wide and shallow, than narrow and deep.For example, I talk with many people who want to measure the daylights out of something like a shopping cart. They measure everything: when someone opens it, when someone closes it, when they add or remove a product, the product size and color, whether they open a pop-up…all this data may be interesting, but it provides them with a false sense of security because the granularity of this data fails to reflect their broader corporate strategy.For example, their focus becomes incredibly myopic and they become obsessed with minute observations like why Firefox visitors convert at a faster rate than those that come in via Internet Explorer. And because they are focused on the weeds, they can’t see that their conversion rate for the broader site has been falling for the past 3 months and they aren’t doing anything about it.In short, trying to “boil the ocean” keeps you from seeing the big picture and operating on your core business goals.

Pitfall #6. Multiple versions of the truth

Analytics success is all about building a baseline for performance (your KPI trend), and trying new things to improve on this baseline. That’s it! That’s why I think it’s easy.  I know other bloggers have argued that analytics is hard, but I’ve done this for a living and I can tell you that it’s not.  Sure, it can be hard – over time – to continue to improve on your baselines.  I’ll grant you that. But that comes after you’ve picked all the low hanging fruit and innovation becomes more critical.  But to be fair, I can’t say I’ve ever met with a company that has picked all the low hanging fruit. So from my perspective, it’s just not that hard.With that in mind, a critical pitfall occurs when customers try to use multiple systems to provide this baseline for performance. In other words, they have multiple versions of the truth.  Most of you know what I mean.

Take a lead generation site for example, like an automotive manufacturer. They will often looks to total number of leads, among other KPI, to determine their success. But they will measure this with 5 different systems. They have leads as measured by Omniture, then leads as measured by their data warehouse, leads as measured by their email system, leads as measured by their ad server, and leads as measured by search. There may be even more systems than that.The problem with this approach is that you get multiple versions of the truth. And you waste your time trying to reconcile all these different systems rather than trying to improve on your baseline.  Now – don’t get me wrong – you must make a best effort to understand why these systems report different results.  I’ve spent countless hours doing this.  And often times when it comes down to it, there are just fundamental differences in the measurement approach (as we talked about in our approach on Data Migration: Fools Erand).  Another example is with clicks.  A “click” as defined by an email provider will likely be different than one provided by Omniture which will be different than that provided by an ad server, which will be different than a “click” reported by a search engine. That’s the reality of the world we live in, so just accept it. You can spend your time trying to solve this academic challenge, or you can spend your time improving your business and beating your competition. You decide.

You must have a single version of the truth, because your ability to optimize your business is based on the relative difference between points on the customer lifecycle. It is not based on the absolute relationship.In other words, if a search engine like Yahoo! says you have 1000 clicks, and another analytics provider puts you at 900 clicks, that doesn’t matter as much as when you compare the 900 clicks as measured by your analytics provider for a search campaign against, say, 700 clicks as measured by the same provider for an email campaign. If you compare numbers from the same provider against each other, you can see how different campaigns are doing in relation to each other. When you compare results from different providers, you’re comparing things not based on the same absolute relationship. Therefore, your results are seriously skewed.

Pitfall #7 – Not Teaching how to fish

In my own analytics career, some of the biggest gains we made from analytics were actually from people outside my analytics team. They came from other business units that we had trained to use analytics. Why? Because they often understood their business questions better than anyone else, so they could innovate the most from the analytics data. In other words, they had the best context for the data.To that end a critical mistake that people often make is not training end users to be self-sufficient. Sure you can send users to programs like Omniture University and conduct internal training. That’s great and a critical first step. But once you’ve done this, you should seek out the people that “get it” – and bring them into your inner circle. When you identify these power users and nurture them, they can become your greatest ally and drive some of the most significant gains you’ll realize from analytics – all without any incremental effort from you and your team. This might sound like a fairy tale – but it works – trust me, I’ve done it and I’ve helped other companies like yours do it.

In Summary

So there you have it. Seven critical pitfalls marketers often fall into when deploying analytics. Granted there are others – but if you go into your next deployment with eyes wide open to these common mistakes – I have every confidence you’ll be more successful than ever before. If you’d like to talk about your unique situation and requirements, we’d be happy to do so. Just give us a call and we can work with you to realign your analytics deployment with your strategic business requirements and industry best practices. It’s what we do best!

It often happens that the person who plans for and oversees the implementation wants to own it and be sole keeper of the data. Whether this stems from a misplaced desire to shield others from any extra work that might come up around the implementation or from a wish to be ruler of the data kingdom, it will cause major problems.Websites touch all facets of an organization, so a web analytics deployment should reflect all facets of the organization. Everyone needs to be involved. This is no longer 1997, when a website was run as a closet initiative.  Not getting requirements nailed down at the outset, from all concerned, can lead to big problems.

Pitfall #2. Focusing on tactical requirements

When deploying an analytics package like Omniture, you must understand your strategic business requirements - that is, what you, as a business owner or stakeholder, will need from the analytics.  What strategic questions do you want to answer? And by strategic I do not mean how popular a link may be on a page…that is tactical.  Strategic questions are directly linked to your company’s strategic initiatives. So start with your company’s 3 or 5 key priorities – as outlined by your CEO – and then determine what questions you need to answer to support those.  Every company I have work at or worked with has had 1000 different things they want to do with their website, but it’s amazing when you apply a lens of strategic CEO goals to these initiatives, how few are actually relevant.  It’s a powerful yet simple approach to prioritizing. For each project, just ask yourself – “how does this support the goal outlined by my CEO?”.  If you can’t clearly answer that question, you’re probably dealing with a tactical requirement and something that is secondary to your deployment.

Pitfall #3 - Believing “data” equals “requirements”

As you go through the strategic requirements gathering process, bear in mind that your requirements are not the same as data.  When asked what your requirements are, your instinct might be to say, “I need hits, I need time spent on page, I need number of visitors…” Those are all metrics, and quite frankly, all three of those are questionable in their strategic value.A requirement is a business question you want to answer.  The metric is the gauge by which you answer that question (aka your Key Performance Indicator).  For example, if you are driving a car, the business question may be “how fast am I driving?”  The implementation of a speedometer (report) allows you to answer this question. And the metric of miles per hour (or kilometers per hour) allows you to determine how fast you are actually going. But just saying you need a miles per hour report doesn’t help because you haven’t established the business question.

So to summarize, understand your mandate at a high level, and then articulate the steps you are taking to hit that goal. If your goal is to improve conversions, there are a number of things you’ll try in order to achieve that: you may redesign your navigation or improve your registration process or make registration optional.  The data you need will fall out naturally from knowing your requirements.

Pitfall #4. Not focusing on KPIs that are specifically tied to the goal.

If you think you have 20 or 30 key performance indicators, then you don’t understand the concept of KPIs.  Think of the dashboard of your car.  There are many data points you can gain by looking at the dashboard, but there are only three or four that really matter.  If you come up with dozens of metrics that you absolutely must have, you’re not focusing on your true business goal. I can guarantee you that.This doesn’t mean that you can’t measure other things that are secondary KPIs - but it does mean you don’t want to focus on them right away.

When implementing an analytics package, above all else you want to ensure that your KPIs are measured.  Don’t waste time on non-strategic measures.  Ask yourself this: if your CEO was stuck on an island and you could tell him only three things about your business so he would know the business was healthy, what would you tell him? If you said the average time spent on a page was 1 minute 30 seconds, that tells him nothing. If you tell him your average revenue per visit was $2.00 and you had 2 million visits, that is something he will understand as a true measure of business success.  There is so much opportunity to measure initiatives and improve on them based on four or five metrics that you can keep yourself busy for months and even years.  Don’t fret about measuring every little last detail. You’ll make yourself crazy and you won’t be supporting your business goals.

My next post will address three more pitfalls. Stay tuned!