Get key stakeholders who are on board with testing in a room together and throw out hypotheses you draw from the data, along with the ideas you’ve debated in circles forever without ever coming to a decisive answer. You can never underestimate how political testing can be in an organization, so make sure you don’t have naysayers in the room on day 1. They’ll come into play later on :)

Once you’ve got all the ideas on the wall, it’s time to start slicing! Let’s say that the list below is what we have to work with.

Here are some key questions you want to ask yourself when evaluating the quality of an idea:

1) What is my hypothesis? - If you don’t have a hypothesis, it’s game over.

2) What are my goals? - Sounds like a no brainer, but I can’t tell you how many times I’ve had clients want to try testing colors or layouts just for the heck of it. You must have a clear goal that you are working towards! It can be more than one goal, such as locating a dealer or requesting a quote, in the case of an automaker, but just make sure you’ve got it clear in your mind so your design aligns with it.

3) How much traffic do I get? - This one’s important. If you only get 50 conversions a day, you will never be able to run a test with 100 different variations. The amount of time it would take to run that test would make it useless. We’re talking months, if not years. To highlight the point: at 50 conversions a day, half of the variations tested wouldn’t even have a single conversion after day 1.

4) Which segments do I want to drill into? This one’s directly related to the amount of traffic you get. Because let’s say you wanted to run a test with 100 different variations, AND you have 4 key segments you’re interested in, such as paid search, organic search, email, and affiliates, well now you are most definitely talking about a test that will take over a year. The point is not that you can’t technically run a test for that long, but as an online marketer competing against other agile marketers, what would be the point?

5) What will I learn if the alternative is the winner? It’s always great to find a winning alternative in a test. But if you just grab 20 different call-to-actions you found on the internet, and It turns out the one from T-Mobile is the winner, what did you learn? Maybe you found that magenta really does rock, but maybe you just stumbled your way onto a winner. The upside of that discovery is nice, but how do you actually build on that learning to evolve your marketing practice?

6) What will I learn if there is no winner? The dreaded lack of statistical confidence…dum dum dum. This one’s going to hurt, no doubt about it. But there are ways to find grace in this result. The first is to make sure you had great answers to all of the questions above. Oh, offering a 20% discount on subscription didn’t increase overall revenue? Well, now we know that we don’t need to cut our price and pricing isn’t as impactful as we expected. Great, let’s move on! Or how about those segments that we set up? If you drill down, you may find that those Mac users really behaved differently than the Windows users, resulting in a cancel-out effect at the top. Now we’ve got 2 separate winners, and more learnings to build upon!

Once you’ve asked yourself these questions and pared down your ideas list, we now need to prioritize what goes first. I like to calculate a score because it forces people to emotionally detach from their favorites.

There are two factors you want to consider when rating a potential test idea:

1) How difficult will this be to implement (both technically and politically)? Again, you cannot underestimate the political impact of a test idea. For example, you just want to test different treatments of women’s clothing on the homepage. That shouldn’t be a problem, right? Well, how do the merchandisers for accessories and children’s clothing feel about that? Or what about changing the link to your sign-up form from small-font text to a big flashing button? The homepage design team might have a problem with that even if it’s technically super-easy to implement. I would recommend saving that test for a later date once the homepage design team has gotten a chance to see the extent of your testing prowess.

2) What is the potential ROI (whether in learnings and/or conversion)? It’s hard to estimate the ROI from a test. If it were easy, we probably wouldn’t need to test. But there are still ways to get closer to a good estimate. For example, testing a page that has a low conversion rate but only 100 visitors a day might be an interesting opportunity to learn, but the resulting revenue bump is probably not going to be impactful. Testing the first page of your checkout funnel could be huge in potential ROI though, even if you’re just talking about 3% lift in revenue per visitor or total orders.

Here’s an example of how you might rate the previous list of ideas.

I think the two factors are actually pretty equal in weight, so I then go ahead and do a straight sum of both to get my idea score. I’ve color-coded and sorted the different tiers of scores here.

A lower score doesn’t mean the idea falls off the board, by any means. It just means that you allow for more time to get that test idea implemented. However, a lower score plus a low value/ROI rating should be grounds for being removed. Hopefully you have a platform that empowers you to run the easy stuff faster and with more control, meaning you’re able to leave IT alone to deal with the hard stuff. We know that makes everybody happier!

The last step is constructing the testing roadmap. Taking that list of ideas we’ve put together, we can now create a much more intelligent roadmap that takes into account the strengths and constraints of your organization, infrastructure and platform. Behold!

This testing roadmap will now inform how you implement your platform and build a knowledge base of learnings and best practices. One piece of advice though, is to stay flexible. This roadmap can and should be fluid, based on what you see in the test results. If you find that testing call-to-actions is yielding significant lift early on, it could be worthwhile to keep digging in that direction for more conversion nuggets. Similarly, if you find that your early attempts at shortening form length really aren’t doing much, you might want to push other tests in that vein further out in the timeline. Ultimately, testing is about listening to your customers, even when they don’t know they’re being asked the question. Don’t forget to remain open enough to truly hear their answers!

If you found this post helpful, re-tweet it with the link or text below! My next post will cover Day 7: We’ve Got Our Testing Roadmap. Now What?

RT @sflily Day 1: I Just Got a Testing Platform. Now What? http://is.gd/mZuf #omniture

Photo Credits:
http://www.flickr.com/photos/martyn/402688339/
http://www.flickr.com/photos/schoolstreet/163727710/
http://www.flickr.com/photos/tomchurchill/335300686/

Twitter has taught me there is a serious segment out there that feels differently. They share the things they watch, read, and buy, and they can start a movement very quickly. For evidence, just read sugarrae’s excellent post about launching tweetwasters. So even if you don’t have much traffic coming in from tweets, blogs, and shares today, it doesn’t mean you might not have a spike tomorrow. Here are some of the things I would set up and test to stay ahead of the game.

Provide Your Own Shortened URLs

The toughest part about testing social media traffic is detecting where the traffic came from. Most URL-shortening services use a 301 redirect when taking visitors to their destination, and that type of redirect cleans up after itself. Unfortunately for us, that means we can’t even detect they came from a shortened URL.

The best solution I can think of is to provide your own shortened URL on the page and make it as easy as possible for your visitors to use it. There are a couple ways to make sure you can track people using your URL. You can use one of the URL-shortening services that provides tracking (good list here from Brian Solis), or you can append a URL parameter to the URL that allows you to track this segment within your analytics. For example, instead of shortening the URL “http://www.omniture.com“, I would shorten “http://www.omniture.com/?src=scl” instead.

If you’re worried about how that might impact your visitors, test it! In addition to the normal conversion event you might assign to a test, I would also look at the incremental traffic received from providing the URL.

Sample Test Design
Audience:
- General
Element(s):
- Hide or Show Shortened URL
Conversion(s):
- Standard conversion event (e.g. add to cart, page views, orders)
- Traffic coming from shortened URL
Segments:
- All Visitors
- Visitors coming from shortened URL (src=scl)

Here’s a screenshot of how that segment would be set up in Test&Target:

Formulate Hypotheses

Let’s assume that you can now track the traffic coming from different outlets like blogs, Twitter, Digg, etc. Based on that, I would try to keep the social ball rolling with the following ideas:

Test whether displaying the ability to share more prominently is effective in increasing traffic. I would imagine that people who followed shared links are much more likely to return the favor.

Does your brand have a presence on Twitter yet? If so, test displaying your most recent tweets in a Twitterstream sidebar.

Do you have product reviews on your site? Jump on the social bandwagon and make sure they’re front and center. Test defaulting to reviews instead of standard product detail copy.

Know which site the traffic is coming from? Reinforce it and serve related content. For example, back in our Offermatica days, MarketingSherpa had a complimentary article about Offermatica so we created a targeted homepage campaign for that specific segment of traffic coming from the article. Through that campaign, we dedicated real estate that highlighted not only the MarketingSherpa mention but also other related articles in the press.

Let’s Talk

Hopefully this post has got you thinking about how you can better leverage the social media traffic coming to your site. For those of you who will be attending our customer Summit in a couple weeks, I’d love to talk more in person about your plans for optimization in 2009! I’ll be presenting with Experian on using Test&Target iteratively to reveal valuable customer insights. Brian Watkins will also be presenting with Forrester’s Jeremiah Owyang and Comcast’s Frank Eliason in a session called “MyFaceLinkSpaceTwitBook: Boosting Conversion with Social Networks”. Jeremiah is a thought leader in the space and I don’t think it’d be an understatement to say that “Comcast Frank” has pioneered the emerging corporate brand presence on Twitter. Oh yeah, and feel free to tweet me at @sflily with your thoughts as well. I aspire to call myself a Twitterholic someday. Hey, a girl can dream…

For more information, check out this great post by Linda Bustos on tracking Twitter links.

1) Agencies Get On Board

Aside from the handful of agencies that specialize in testing and conversion optimization, I’ve seen far less progress than I expected this year. To do a sanity check, I googled “[agency name] multivariate” on the top 4 digital agencies of 2007 from AdAge. Here’s what I found:

Avenue A | Razorfish is the only search that yields any substantial result, and that is to a partner’s press release focusing on the platform rather than their own in-house expertise. Contrast that with searching for “[agency name] creative”, and you’ll find that Digitas and Ogilvy rank #1 in SEO and Avenue A | Razorfish at #3.

I know that part of the problem is that agencies have a tough time convincing clients to test. But I suspect that agencies are also not yet accountable for testing, so why deliver some options that may quantifiably fail when you can deliver a single option that unquantifiably wins? The latter requires less level-of-effort and chance of hassle. Unfortunately it also has less upside for the customer. Hopefully agencies will consider the significance of providing more value to their customers, and what that may yield for them long-term.

My reason for wanting agencies to get on board is selfish. I believe that agencies (at their best) are the strategic partners of customers, and they can help prove the value of testing. But it feels a bit like the chicken-and-the-egg. Agencies aren’t compelled to actively bring testing skills and experience in-house without clients asking for it, and clients may not be educated enough to take on testing without an agency resource!

2) Offsite + Onsite = 1 Visitor Experience

I’ve definitely seen great strides already taken this year in trying to treat a visitor’s path through offsite and onsite as one cohesive experience. Some of our customers are doing really interesting projects involving the testing of retargeted ads out on 3rd-party sites and building a single profile that can be extended and reinforced at every online touch point. I’m excited to see how that plays out in 2009.

However, I’ve also seen a lot of poorly executed landing page experiences that show there’s no discussion going on between those in charge of acquisition and those in charge of the site. If there is any confusion around where to begin testing, I’d highly recommend starting there. Landing pages are typically easier to change and tag, and transforming a post-click experience from general and 80% irrelevant to targeted and 100% relevant usually yields great lift!

3) Multivariate Testing (MVT) Becomes Another Tool in the Toolbox

There’s no question that multivariate testing is an incredibly powerful tool. But it’s certainly not the only tool you need or have to increase conversion on your site. I think of it as the hammer in the saying, “when all you have is a hammer, everything looks like a nail.” There are many efforts that multivariate testing is not the best tool for, including massive functionality changes (e.g. 5-page checkout vs. 1-page checkout) and automated behavioral targeting. The problem with looking at MVT as the all-purpose cleaner is that it limits your vision to consider other types of tests, and it also reinforces a fragmented marketing division. I can’t count how many meetings I’ve been to where the catalyst is an RFP for MVT. Once we start talking about the site’s initiatives though, we begin to understand that there is also interest around segmentation and targeting. However, that’s a separate project manager and a separate RFP, and nobody is interested in getting together in the same room.

4) Targeting Belongs in the Same Toolbox

Targeting for the sake of targeting is not always effective. I have seen numerous cases where the first attempt strikes out. Sometimes people don’t want to see their first name read back to them, but they do like seeing their search terms repeated. Others like to see an offer reinforcing the deal they saw in the ad, but they may not want to see the same exact image. Without testing, you’re still not making data-driven decisions. At best, you’re making more educated guesses.

5) Web Optimization Collaboration Takes Off

Are you sensing a trend here? : I want more collaboration between acquisition marketers and site marketers, companies and agencies, vendors and companies! I think everybody wins when we share more of our testing experience and I’m convinced there has to be a way to do it without giving up competitive intelligence. So far the weboptimization group has had some good discussion around getting testing buy-in, and some very spicy discussion around vendor comparisons. Tell me how you think a forum should be constructed to be most beneficial, and then let’s work together to make it a reality.

6) Succeed Often and Fail Fast

This one’s a little off the web optimization rails, but I wish President Obama all the best in making 2009 a year of recovery and successes. I don’t think it’s possible to not fail, especially given these tumultuous and uncertain times, so I’d just like to encourage him and all the rest of us to fail fast and keep on truckin’!

Happy New Year!

Photo credits:
http://www.flickr.com/photos/jolienvallins/1505871497/
http://www.flickr.com/photos/down_under_images/679400938/

Should Testing Be Outsourced?

That last segment is the one I grapple with the most. I don’t think the question is whether you can outsource testing and optimization, but whether you should. For any company serious about their online channel, I believe that their marketers should already be thinking every day about how to improve the site and all acquisition efforts leading to the site. Those ideas should be the fuel powering the optimization engine, so what would make a company want to turn that over to somebody else to manage? Does that mean there’s no fuel in-house, or that the fuel is bad grade? If so, that seems to point to a larger problem for the company than just whether testing can be outsourced.

None of this is to say that I don’t believe in the value of vendors, consultants and agencies contributing to your optimization efforts. I’ve seen many cases where 3rd-parties have helped produce fantastic results for their clients. They bring their extensive experience with other companies and fresh insights to help marketers think about their decisions from a testing perspective. But I’ve also seen companies slash those parties off the budget without thinking twice because nobody at the company actually cared about testing. Here are the main reasons why I don’t think testing can be fully outsourced:

1) No Testing Culture - There is a huge unquantifiable benefit to having a testing culture instilled at a company. It means that people are more thoughtful and less careless because they are forced to make data-driven decisions. It also means that they feel more empowered to have ideas because they will get to see them in action on the site. An employee who sees that he or she has made hundreds of thousands of dollars for their company through a great idea that got tested on the site will certainly be motivated and motivate others to produce more stellar results!

2) Less Learnings - If a 3rd-party is managing a company’s testing efforts, then all the learnings that come out of that testing don’t usually make it back out to the rest of team, especially if those tests produce negative ROI. They are still incredibly valuable though, as they can inform how designs should be planned moving forward. Imagine if an agency tests a big hero image against several small product images and finds one much more compelling than the other. That could change how other marketing efforts such as site redesigns and direct mail pieces are designed, but only if that information is circulated.

3) Easily Removed - As I said before, divorcing optimization from the company makes it easily removable as a nice-to-have instead of a must-have. This issue is not isolated to just outsourcing the operation though. I’ve also seen it happen many times when there is only one person at a company managing testing. That champion might be generating many testing successes, but once they leave, nobody knows how to run it or wants to take on extra work. Beware of tribal knowledge! The most successful testing companies maintain a knowledgebase so they can easily reference previous learnings and not be back at square 1 if a subject matter expert leaves.

4) Less Collaboration - Testing forces people to talk to each other. Disconnected landing page experiences are often a result of disconnected marketers. An acquisition marketer is incented to get people to the landing page. A site marketer is incented to drive people to conversion. There needs to be some common objective that forces them to talk to each other. I’ve also seen testing help align marketing and IT. Instead of asking IT to simply put another tag on the page, marketing should explain what that tag’s purpose is and that it is a 1-time effort that will actually make their lives easier moving forward. That helps IT understand the value of implementation and provides context for why it’s different from the other tags on the page.

Again, there are amazing resources out there to help you start testing successfully. This post is not meant to detract from the immediate value they offer to jumpstart your optimization program. Take advantage of the best practices learned by others and use the valuable strategy, advice and creativity they can provide. But please, consider the benefits of using 3rd-parties to get your testing legs under you so that you can run on your own eventually! Spread the love and bring people into the optimization process. It’s more fun to have a running partner than to watch other people running laps around you.

Photos credits:
http://www.flickr.com/photos/cindy47452/144655993/
http://www.flickr.com/photos/avalonstar/163502744/

Testing is inherently collaborative. It feeds off the behaviors of your visitors, but it also requires coordination among multiple groups for successful execution, including marketing, creative, and development. Those who begin testing for the first time can truly benefit from the experiences of those who went before them. So it begs the question - why isn’t there more collaboration going on out there?

I could hypothesize that it’s because testing is still a competitive advantage at this point. I certainly have some clients who feel this way and are very protective of their test designs and results. But I believe they’re in the minority.

The best conversation I’ve read about testing in the last year took place in Avinash Kaushik’s blog comments recently. And while I love all the responses the post stirred up, I can’t help thinking that there’s got to be a better place to keep the conversation going.

Toward that end, I’ve created a group focused on web optimization where I’m hoping a lot more collaboration will take place. This is a group for marketers interested in or currently testing and optimizing their sites today, regardless of platform or vendor. I invite companies, agencies, and vendors to join the conversation.

I’ll most likely be creating discussions there around best practices and adding a lot of the questions I commonly get around testing. Even though it’s fun to blog here, I get a little lonely banging around on my laptop alone. I’d much rather get more feedback from those of you on the front line and I’m hoping we can all find ways to help each other as the testing community continues to grow.

I encourage anybody interested in testing and optimization to join the group today and participate! Participation is key.

Group: http://groups.google.com/group/weboptimization

That’s really encouraging news, especially in a downturn economy. It shows that marketers are realizing that testing is one of the few strategies that truly enable you to do more with less. Instead of having to invest linear amounts of money to acquire traffic, you can instead invest your ideas and hypotheses to increase conversion and revenue on your site.

I often find that the Q&A is the most interesting and engaging part of a session. It’s the closest we get to understanding what’s really on the minds of marketers today, and what they were hoping to learn from those of us on the panel. One of the questions that stuck in my mind was, “What are good strategies for optimizing social media traffic going to landing pages?”

To be honest, I didn’t have a great answer because I haven’t seen many tests involving social media traffic. Maybe it’s because we don’t have customers who are investing much in social media yet. But I thought about it more on the plane, and then at home, and I think the answer lies in going back to the standard best practices of first defining what your success event will be, and then forming a hypothesis around how to improve it.

For example, if you told me that you ran a content site and wanted to increase page views, I’d say that those who come from social media probably have a higher likelihood of sharing content with friends. My test idea might be to test promoting sharing links higher up on the page, or repeatedly throughout the article. But if you told me that you ran a retail site and you wanted to increase orders, I might try to promote highly-rated products based on the hypothesis that social media visitors place a greater premium on the opinions of peers.

Another question I got was, “Okay, you talk a lot about testing, but how do I test on my site? Do you sell a product?” Yikes, the dreaded vendor pitch trap! I have to admit there was a bit of spastic product naming as I tried to give a very neutral answer. I struggle quite a bit with the balance of giving information and opinions without selling any specific platform, both in places like PubCon and here in this blog. I know that you, as the reader, have come here knowing that it’s a corporate blog run by Omniture, but I also think that you’re more interested in learning about marketing principles and customer insights than about what we’re selling. So I err on the side of avoiding the product conversation completely, to the detriment of those who are already customers. Luckily, my very capable colleague, Brian Hawkins, has begun a blog recently about the ins and outs of Omniture Test&Target, so I would direct readers there for specifics on how to implement many of the examples and case studies we show.

The last question I have is for you, the marketer who attends conferences like PubCon. When you go to a session about landing page optimization, what are you hoping to hear? Are you looking for tactical advice, or customer case studies, or market research findings? Would you rather see a presentation that tells you about the latest marketing strategies and where we see the market going, or the one that lays out 10 easy tips to try on your site today?

One Site, Many Goals

These are the 3 major objectives I think a campaign site has to accomplish:

1) inform
2) persuade
3) solicit

The problem is that not all visitors progress through each stage, nor do they follow them in any particular order. It’s a bit of a chicken and the egg…and, uh, the cow problem. Luckily, it’s also the perfect scenario to bring your acquisition marketers and site marketers together. Your acquisition channels can help segment your visitors by goal and communicate that back to the site. More on that later through real examples from the Obama and McCain campaigns…

Time = Voters

What I find interesting about political campaigns is the urgency that drives execution at a much faster pace than most companies out there can accomplish today. I suspect that the marketing team behind each candidate ranks near the top for agility and speed because there simply is no other option. The primary problem is not whether there has been enough budget and headcount allocated for 2009, it’s about who bid on the right keywords representing the issues voters are searching for today and who can react fastest to the latest event to hit the headlines.

As we’ve all seen over the last few months, advertising is one of the major vehicles driving a campaign’s messaging and branding. It shows us which topics candidates are focusing on and which channels they are spending the most money on. What’s sometimes missed, though, is what that advertising then tells them about us. There are few advertisements that don’t ask you to do something in return, whether it’s donating on a website or joining a mailing list. Below is a comparison of each campaign’s ad on the same day in the same location, along with the corresponding landing page.

The McCain campaign’s ad on Drudge Report:

Ad’s Landing Page:

Obama Campaign’s Ad on Drudge Report:

Ad’s Landing Page:

You’ll notice that there is a widget available to input information about your income so that you can calculate how much you could potentially save under the Obama/Biden tax plan.

Connect the Offsite and Onsite Experience

Could these two experiences be any more different? McCain’s ad is a straightforward call-to-action to donate. It focuses singularly on the “solicit” goal. However, there are key elements of that ad that are lost once the visitor lands on the site. For example, what does it mean to “Invest in Victory”? Should we at least see that headline reinforced on the landing page, if not elaborated on with how individual contributions can help?

It’s also interesting to note that the call-to-action specifically uses $50 as the amount to contribute, but it is neither highlighted nor pre-selected once the form to donate is presented. I would imagine that some market analysis led to the emphasis of $50 in the ad, so why not continue that through the experience by making the process to donate $50 just a little bit simpler.

In contrast, Obama’s ad is targeting voters based on the issue of taxes. It takes the approach of focusing on the “inform” and “persuade” goals. You’ll see that the landing page calculator asks for anonymous information and then returns an estimate of savings. Here’s what the page looks like after filling out the calculator:

Note that the calculator has changed into a form that now asks for my information to keep in touch. We’re now moving into the next stage of solicitation.

Engage the Visitor

From a design, flow, and relevancy perspective, I think this ad to landing page experience is very well-executed. Beyond that though, I think it’s also a clever way to get a lot of information about your visitors in a way that is not explicit. Let’s take a closer look at what we learn as the visitor progresses through the ad and landing page.

After the visitor clicks on the ad:
We can infer that this visitor is interested in learning more about taxes and saving money.

After the visitor fills out the calculator:
At a minimum, we know the visitor’s annual income and filing status. On top of that, we may know how many dependents they have, whether they are over 65, if they have child care expenses, if they are saving for retirement, the outstanding balance of their mortgage, and if they have college expenses. That is A LOT of information when you look at it from a profile targeting and segmentation perspective. Imagine how much more relevant and engaging you can make this visitor’s experience across multiple channels including display, email, and site by having this type of data.

After the visitor fills out the “Stay Informed” form:
We now have the visitor’s email address and zip code to tie all the information together in the database.

To recap, here’s the wealth of information that the Obama campaign has in their database now after a visitor clicks one ad and fills out two short forms:

• email address
• zip code
• issue interest
• income bracket
• filing status
• age*
• # of dependents*
• mortgage balance*
• saving for retirement*

*optional field

Would you send a different email to the voter in a swing state making $40,000 a year vs. the voter in New York making $300,000 a year?

These Principles Apply to More Than Politics

The strategies employed here are not unique to politics. They might just require a shift in how you approach your business and define the goals of your acquisition marketing efforts.

For example, let’s say you’re a retailer that sells HDTVs. Instead of an ad promoting a specific model, maybe you show an ad that provides information on how to select the right HDTV for you. The landing page then asks the visitor about their budget, preferred size, and other relevant parameters. Maybe that then leads them to personalized search results, or you display another short form that enables them to receive alerts when an HDTV goes on sale within their budget.

Think about how you can make each touch point with your visitor more relevant and engaging so you can draw them through a cohesive experience that both leads them to the water and increases your bottom line. In other words, ask not what your visitor can do for you, but what you can do for your visitor. And don’t forget to vote!

Hitting a Roadblock

It was hard to get people to talk about the reasons why during our session, but I spoke to these marketers afterwards, and the overriding sense I got was that they were completely handcuffed and roadblocked by processes and politics. No one felt like they actually owned the site they were supposed to be marketing on. Changing content on the site, let alone testing it, felt like an insurmountable challenge.

I realized that I could show as many examples of bad landing page experiences, wasted opportunities for testing and targeting, and case study successes as I wanted, but the problem these marketers faced was not a lack of belief in testing - it was the inability to sell this belief up the chain. With that in mind, I want to help arm you with an arsenal of reasons why your company can’t afford to delay testing another year.

1) Testing Drives Results

ZeroDash1 conducted an independent study at the eMetrics show in San Francisco this year, and one of the key questions they asked was whether marketers who were testing thought it was worth it. Take a look at the results. (Note: this survey was not solicited by Omniture in any way)

92% saw positive ROI! How many other initiatives can you say that about? It also makes me really proud to point out that 54% of those who say they “can’t live without it” use Omniture Test&Target. As part of a company that has been evangelizing testing for the last 5 years, it’s incredibly rewarding to see affirmation of the original bet that Offermatica’s founders made - to make a product designed for marketers, not statisticians or IT.

2) Optimizing On-Site Increases Off-site ROAS

I recently surveyed the web to see the state of landing pages today. What I found was that companies are still missing out on so many opportunities to increase relevancy and engagement on their sites. I’m only going to show one example here, but it certainly reflects the majority of pages I visited.

Note the ad at the top of the page. It’s a timely ad considering we all have savings on the mind these days. Here’s the page I landed on when I clicked through.

Was that what you were expecting? I was looking for more sale information, a preview of those thousands of items, the hundreds in savings. Instead, I got sent to the lighting page. Do I think that Design Within Reach’s marketing team thought the lighting page was the best place to send this traffic? Nope. I think the media marketers probably weren’t talking to the site marketers, and even if they were, the site marketers probably couldn’t change anything about the page or put up a new sale preview page in time. This example represents a huge missed opportunity though. Imagine the difference in bounce rate if a user had landed on a sale preview page instead.

3) Hope is Not a Strategy

In the same ZeroDash1 survey conducted at eMetrics, marketers were asked whether their pre-test hypotheses were always correct. “100% of survey respondents admit that their pre test hypotheses are not always correct.” 100% means not one single person can say he or she has always been right. Even though it sounds like a no-brainer, it means that anybody who is not testing today has been wrong at least once about a new idea or design that’s been pushed out to production. Testing allows you to understand whether your ideas actually impact your success metrics and then make data-driven decisions instead of hope-driven leaps of faith.

Put another way, optimizing your site enables you to fail fast. One of the marketers in our session said that some of his most significant test results are those that show him which ideas don’t succeed. To know that something doesn’t work on your site allows you to put that debate to bed and move on to the other ideas on the list.

4) Time is of the Essence

The window of time to get a head start on the competition is closing quickly. 69% of marketers who have not already begun testing plan on implementing it within the next year.

If you look at the companies who are producing the most innovative products and sites, you’ll notice they all have one thing in common - they are ahead of the curve. When Amazon created a retail site, they didn’t wait until their competitors figured out that whole personalization thing first. When Southwest Airlines came up with a new way to ticket and board people, they didn’t wait to see whether the process would fail in another airline’s hands first.

Optimizing your site is a way to innovate. Understanding how your customers react to content and layout and design gives you a huge advantage. Linking their offsite experiences with their onsite experiences allows you to reach out across multiple channels. That integration is not only an opportunity to increase revenue and conversion, but also a way to present a cohesive message about your brand.

I have a very successful customer who I’ve been trying to get out to conferences to talk about the stellar test results he’s seen and the ways in which he went about installing a testing culture at his company. He’s hesitant to talk about anything related to his optimization efforts though. He realizes he’s found a competitive edge at a time when there are few edges to be found, and now is the time for him to exploit it for all it’s worth.

5) Arm Yourself With Data

You probably have analytics running on your site today that can help you identify problem areas, places where your visitors are bouncing at a high rate or clicking away from the purchase path. Think about what you could test in those areas to improve your success metrics, and then bring both the data and the correlating ideas with you.

Have you considered segmenting your audience as well? Make the case for targeting large segments intelligently. The segments that I would recommend examining include traffic source (email, PPC, organic search, banner ads, etc), search queries (branded vs. non-branded), time of day (work hours vs. non-work hours), day of week (weekday vs. weekend), and loyalty (new vs. return vs. past purchase history). Bring data that is specific to your company, your business, and your competitors.

Lastly, bring case studies that show other companies deriving value from testing and targeting on their sites. If you need any inspiration, check out Omniture Test&Target’s own repository. Here are a few specific Test&Target ones to check out:

• Dallas Morning News’ Subscription Optimization
• BabyCenter’s Landing Page Optimization
• Musician’s Friends’s Affiliate Reinforcement
• CNET’s Engagement Opmitization

Implementing a new culture of testing and data-driven decision-making does require changes in your organization, there’s just no way around it. It doesn’t happen overnight and it doesn’t happen without someone to enthusiastically support it, preferably someone high up on the food chain. But let’s also recognize that while change can be scary and political, it is also necessary for any company trying to keep up online today. And hey, the goal is not to keep up, the goal is to win. We’ve seen many of our optimization champions go on to be recognized as rock stars within their companies, people who can point to data and attribute annualized gains in the millions back to their specific testing efforts. Is that something you can say about yourself today?

1) Testing - the process of testing alternatives on your site to see what converts best. Testing is most often implemented as simultaneously delivering alternative content to your visitors in a randomized fashion.

2) Targeting - the process of displaying relevant content to a particular segment or individual based on implicit and explicit variables such as gender, search terms, and intent. Targeting is most often implemented as delivering content to your visitors in a rules-based fashion. For some companies with enough site traffic, targeting can be delivered to the individual in an automated fashion.

Testing and Targeting: 1+1=3

Both efforts rely on the ability to deliver content dynamically on your site. Both are designed to increase key success metrics on your site. However, they are often separated as two completely different initiatives within a marketing organization. While people seem to be interested in the idea of combining them, they are usually already siloed as different projects with different teams, budgets and timelines by the time we have the discussion. I think bringing testing and targeting together is really one of those cases where 1+1=3 though.

Let’s first talk about testing by itself. It’s a great way to figure out what version your visitors prefer. If you’re just throwing new designs up on your site today on a wish and a prayer, testing is definitely a must. But then think about who your visitors are. Are they all people who are coming for the first time? Are they all visiting during work hours? Do they all enter on the same page looking for the same thing?

If your answer is no to any of these questions, then you have to ask yourself if it makes sense to assume that they all prefer the same winner in any given test. The easiest way to figure this out is to segment your reporting. That doesn’t change anything about how the test is designed or implemented, but it can give you additional learnings in the reporting and analysis.

What if you found out that the reason a particular version won was because it resonated deeply with your new visitors coming organically to your site? And balancing out that great lift was actually a negative result with other smaller segments such as direct mail and branded queries coming from paid search? This type of data is incredibly valuable for any organization trying to get deeper insights into what their visitors are looking for and responding to. What makes it invaluable is to then take action on those learnings by delivering that targeted, winning experience to your organic, first-time visitors and then simultaneously beginning a new test on the rest of the traffic.

eHarmony ran a test with us to understand whether adding tabbed navigation to a landing page would be more effective in getting users to convert. The hypothesis was that providing more information about the service directly on the landing page would be a positive change. Here are the different versions.

What we found was that the test actually performed worse for overall traffic, and that people preferred the simplified page without the AJAX navigation. Once we dug into our segments though, we found a more complex and interesting story. We had set up geo-segments to track how people from different countries behaved, and it turned out that visitors from Canada significantly preferred the navigation. This learning would never have been found without segmentation, and worse, the tabbed navigation would have probably been thrown out based on the overall negative lift.

What is Targeting Without Testing?

What is targeting without testing though? Isn’t it the same as just putting your hypothesis out onto the site and hoping your hunch aligns with your visitors? For example, let’s say you own a retail clothing site, and you want to target visitors who look at accessories and then return to the home page. You may wonder which accessories to now reinforce upon their return, but what if you should instead be asking yourself whether accessories is the right category to show? What if you should be featuring women’s clothing instead because there is a high correlation between the two or the potential for higher margin profit? Or what if showing 2 featured products instead of 6 would make all the difference? Do you know which price range you should be staying between? All of these questions are great opportunities to combine testing and targeting into a single initiative focused on creating conversion and revenue on your site.

BabyCenter ran a test with Test&Target to understand the impact of increasing relevancy on one of their most highly-trafficked keywords: “baby names.” Below is the control version, along with the alternatives tested against it. What’s interesting to note here is that all of the alternatives have some element of targeting in them. Recipe B offers some relevant text, Recipe C adds some category links, Recipe D provides top 5 names of 2005 for both girls and boys, and Recipe E builds on Recipe B with a few more bullets of information. Can you guess which one won?

If you guessed Recipe D, then you are in sync with most of the marketers who see this case study. However, the answer is Recipe B, the alternative that provides the simplest reinforcement of just a headline and a couple lines of copy. Here’s the test data:

You can see that Recipe B provided 65.32% lift against control with very high statistical confidence. In fact, nearly every alternative performed better than the control except for Recipe D, the version that people most often pick as the winner. It makes sense that we naturally gravitate to Recipe D because it provides us with the most relevant information given that we are searching on the term, “baby names.” However, since we are measuring the success of the landing page based on those who sign up for the magazine, we have to balance driving both relevancy and engagement. Without testing this targeting effort though, we would have no idea whether we picked the right mix.

If you are in the process of testing today, I’d strongly advocate setting up segments so you can begin to see which ones behave differently. Targeting is a natural extension of testing, but don’t forget to go back and test your hunches as well, whether they apply to one segment or all of your traffic. Taking action on your data is the best way to do more with less, a strategy we will all have to follow more closely through the immediate future.

1) The Internet changes every day, whether it’s based on oil prices, local and global news, or your competition switching tactics and prices.

2) What worked on your site six months ago may not be the same thing that works today, and will most likely not be what works six months from now.

3) The most successful Internet marketers are light on their feet - agile, flexible, and able to adapt quickly.

4) There is no magic formula to the Internet. We cannot say that if A & B, then C will happen each and every time.

In a perfect theoretical world where none of the statements above were true, I would run full-factorial each and every time. That way, I get to understand which exact combination is best out of all the possible combinations of elements, and even calculate all the different levels of interaction between elements. However, we unfortunately don’t have the gift of infinite time when running tests and analyzing results.

I recently read a case study touting full-factorial and the 576 different combinations tested. It had great graphs and charts of data, but, in my opinion, there were 2 huge things missing:

1) How long did this test take to run? If I go by Google’s handy calculator, I would estimate it took nearly half a year:

I don’t know many companies who have the luxury of running a test longer than one month, let alone five months!

2) How did different customer segments perform? Were segments even set up and tracked? With 576 combinations to test, even setting up two coarse segments such as new visitor and return visitor would double the amount of time the test had to run. In this case, we’re now looking at closer to a year! How can any company with various acquisition points and customer behavioral segments run a test and not slice their population up to understand where the differentiation lies though? Consider the customers who search on “guitar center” vs. those who click a PPC ad after searching for “les paul guitar” - is it possible they might react differently in a test? I would say it’s quite likely.

Does all this mean there are no cases where full-factorial might be more effective? Not at all. I have recommended running a full-factorial to clients in the past when the elements they were testing were highly graphical and seemed interdependent. Take, for example, a row of different photo categories (Abstracts, People, Close ups, B&W, etc) to choose from where each category’s photo representation would be considered an element to test. That seems like the appropriate place to run a full-factorial because you may not want 2 pictures that look very similar to appear side-by-side. However, there are trade-offs to dedicating the time and traffic to full-factorial. You most likely have to severely limit the number of elements you will be testing at once. You may also have to forgo customer segmentation unless you are one of the few companies with the benefit of millions of visitors a day.

I think that one of our own customers actually summed it up best for me last week. John Pace, a true champion of testing and the head of optimization at Real Networks, likened fractional-factorial testing to a barometer. He’s a sailing man, so forgive me if the analogy doesn’t sync up for you. A barometer measures atmospheric pressure, but its value is not so much in the precise measurement as the notification that there is a directional change in pressure.

In much the same way, testing is supposed to give you directional feedback on what is performing and resonating best with your visitors. Testing is not a document or proof that you can use to be 100% sure of how your visitors will behave moving forward. Because of that, I question how valuable it is to spend 5 months running 1 single test for learnings that may no longer be applicable by the time the test has completed and the data pumped through analysis. Instead, why not take the winnings and learnings of your week-long fractional-factorial multivariate test and then run another test that builds off that new and improved baseline. If you can approach your testing program that way, I’m confident that you’ll find more upside in both lift and learnings in the same 5-month period.

At the end of the day, no matter which methodology you end up using, the race for conversions and revenue is not going to be won by the might of your statistics. Your strengths should be creativity, innovation, and the ability to listen and react to your customer. Employing those in testing will get you to the finish line; it’s just a matter of whether you get there sooner or later than the rest of the pack.

Definitions

Multivariate Test: A multivariate (MVT) test enables you to test multiple elements simultaneously. A multivariate test example would be to test the banner, headline, copy, and call-to-action on a landing page. The benefits of running a multivariate test are that you can test more elements at once than an A/B test, and you also get information about which elements were most significant and which alternatives produced the most lift.

Full-Factorial Design: Full-factorial design tests all of the different combinations of elements and their alternatives. For example, if you had 7 elements on a page with 2 alternatives each, a full-factorial design would test all 128 (2^7) combinations.

Fractional-Factorial Design: Fractional-factorial design tests a subset of all the different combinations of elements and their alternatives. In the same test example above, a fractional- factorial design using the Taguchi method would test 8 combinations.