By now I’m sure you have a good idea of how Test&Target will help you test out design ideas, conversion funnels, and landing pages. But those are all user experiences that take place on your website. When you are doing acquisition marketing or cultivating value from a database of customer email addresses the stakes are pretty high. You are either paying for the display ad impression or you are paying to send them a message that will hopefully get them to click through and come to your site to make a purchase, complete a lead capture form, or perform some other sort of success (eg: for firms such as auto companies it may be to build & price a car or request a dealer quote).

Whatever the end goal, it’s costing you money to get your message in front of them. And I’ve seen it proven time and again, and you probably have too, that the offer and the creative execution of the ad or email makes a big difference in response rates. It can also make a big difference in whether those who respond are qualified – meaning they actually are interested in the offer and could become a valuable customer. Why not test out the offer and the creative that is being put in front of them the same way you might test an offer and creative of a homepage hero banner on your website? It’s bound to make your spend (or send) more effective.

Here’s how it works: the Test&Target product has a feature called “AdBoxing” that allows you to create a testing script that doesn’t require JavaScript to execute the testing campaign (fyi: you can also use Flashboxing or other Adobe products/services to achieve a similar result). Instead, it produces a simple URL that points to Test&Target – and wherever you place this URL, it will display content A or content B. So if you want to a/b test two different display banners against each other and see which one brings in the most traffic, revenue and orders, you just place banner A into one Test&Target recipe and banner B into another. Create an AdBox URL and give that URL to your ad network. When your AdBox URL gets published, it tests banner A against banner B and lets you monitor how they are performing against each other — based on actual conversions that resulted on your site for each banner, not just response rate.

One of the largest advertisers in Latin America decided to take this a step further and use a combination of AdBoxing with landing page optimization to find the ideal combination of ad + landing page (see Figure 1). As I’ve said before, this is the “least leaky bucket” approach.
Figure 1:

A similar concept applies to email sends, but can be made even more powerful by pairing Test&Target with an integrated email provider such as ExactTarget.

Often when you push an email send to a segment of your customers, there are debates over what the weekly offer should be to that segment and what the creative treatment of the email should be. Why not test those things and ease the debate, while making more money in the process? That’s the principle behind the Test&Target integration with ExactTarget. First, you push the send to your customer segment with both version A and version B split randomly among the customers in the segment. Half of the recipients get version A, half get version B. But as customers begin to open the email and respond (or ignore) the offer, Test&Target is learning which version is performing better — again, performance is judged not just on response rates but on actual resulting conversions on your site. Pretty soon you have a winning version of the email, and the system can change all of the emails that are still sitting unopened in customers’ inboxes so that when they are opened they will all have the winning version – no more testing needed for this send.

This same application of testing can be used to squeeze more value out of each send – even long after the day of the send. For example, what if you send an email that has a time-sensitive offer (like a President’s Day Sale)? By the time the sale has come and gone, there’s likely a large percentage of emails promoting the sale that are still sitting unopened in customers’ inboxes. When they open them, if it’s an expired sale then that’s a wasted opportunity to market to that individual and it’s a poor experience for them. It’s a throwaway email from you. So after the President’s Day Sale is over, why not use Test&Target to change out the offer in any email that is still sitting unopened in a customer’s inbox? Replace it with an offer that is current, and you’ll squeeze some more life out of that old send after all!

Best of luck with all your efforts in testing to improve your digital marketing, increase conversion and revenue, and improve the experiences your customers have with your brand. I hope this series of blog posts has helped spark some ideas for you, and that you’ll roll up your sleeves and go well beyond the concepts I’ve written here!

-Brig Graff

At some point, though, you will likely find that your curiosity is just buzzing to join up your analytics efforts with your testing. And that’s time to test #4: when you spot an opportunity in your analytics.

Here’s an example I performed for a large online retailer that was wondering which products it should consider testing out in its home page hero banners. Rather than just running a test of the top revenue products, wouldn’t it be good to pinpoint some key products that have statistical indicators of being more likely to win the test? I thought it would be. So here’s what we decided to do…

We took the 50 or so top-selling products and placed them on a scatter plot, with Conversion Rate as the x-axis and Avg. Selling Price as the y-axis. You would normally expect that as price goes up, conversion goes down because it takes relatively longer (ie: more visits) to research and consider a high-priced item before purchasing. You can fit a trend line to the center of that scatter plot and see that relationship quite clearly.

Figure 1:

However, we decided to go out two standard deviations from that center trend line and set boundaries - which by definition include 95% of results. This allows us to see the 2.5% of products that statistically outperform the pack on these two metrics (in green) as well as the 2.5% that statistically underperform the pack (in red).
The conclusion? We should look at the products that outperform and ask if we have tested merchandising them prominently throughout the site and in featured advertisements. If not, we probably should be. And those in red? At least we know that we probably don’t need to start with those when picking products to test in promotional banners on the homepage.

Some would say that you need to have a breadth of pricing options on the homepage though - you can’t just showcase $1,000 items all over. You need some $50, $100, and $500 items. That’s fine — but don’t make it those two on the bottom left. Move horizontally to the right from those and you’ll hit some other product that has a similar price point but a more acceptable conversion rate.
So, what did we find when we looked at which products these four were? Surprisingly, the green items were not being promoted much at all and the red ones were at the top of their category page. On top of that, the green items actually had not just a higher price+conversion combo, they had a much higher gross margin than the red ones. That doesn’t mean the green ones are a slam dunk - but it’s definitely an indicator that you ought to test them out in a few more prominent promotional locations throughout the site and in your advertising - see how they do for you.

Often this reluctance to test entire funnels stems from the fact that IT will have to be much more closely involved since the technology may touch an order management system. A campaign landing page usually does enjoy the luxury of not having to interface with those systems or require IT involvement – marketing’s front-end designers can usually execute a landing page test.

Let’s look at an example, and how to tackle it very easily. A large financial services company had long wanted to decrease the abandonment of its online credit card application form. It was several pages long, with a series of questions and form fields, almost all of which were necessary in order to process the application. The response would be that the individual was approved, not approved, or that a decision was pending further research.

One of the company’s developers had explored building an AJAX-based form that would reduce the process to a single page that built as information was submitted. However, debates continued to abound within the company as to whether a multi-page form would convert better or the single-page form. The answer? Test it. But how?

Assuming you have a developer to build the new conversion funnel for you, the process to test it is very simple: Build the proposed new application form, and at the beginning of the existing funnel create a straight-forward redirect test. Send half of your customers to the first page of the multi-page form, and half to the 1-page AJAX form. Upon submitting the form, all customers get sent to the same ‘thank you’ page where you record the conversion. Now you can view the comparative performance of the two funnels against each other quite easily, just like a simple a/b test of a landing page.

But what if your credit card application funnel (or shopping cart checkout, or lead capture form) ends up showing no signal between such radically different experiences? What if the results appear to be the same, even long after statistical significance has been reached?

That’s when you perform customer segmentation analysis on the results. You may find that customers with fast connection speeds love the 1-page AJAX form just as much as your customers with slow connection speeds hate it. At that point, rather than choose one experience and push it out to everyone you should maximize your aggregate conversion rate by setting up a targeting rule where you give the 1-page form to anyone on fast connection speeds and the multi-page form to anyone on a slow connection speed. This way you don’t just get a so-so conversion rate overall. You get the best possible conversion rate from both groups.
Good general rule to follow: Instead of delivering the best-performing experience to everyone, deliver the best-performing experience for everyone. We’ll talk more about targeting another day…

Resolving internal disputes is actually one of the most common reasons I see companies purchasing a testing solution. Somebody has an idea that the article detail page on their media website would help keep customers on the site longer and consume more ads if it just were re-designed in X or Y way. Somebody else hates the new design idea, and thinks it ought to be kept as-is. Yet a third person asserts that both designs are wrong, and she has her own design that she claims is better than either of the other two.

This debate can go on endlessly — for years, literally. Everyone has an opinion…and they are very willing to back up their opinion with lots of squishy assertions, like “everyone hates Flash,” or “we need an extremely bold call-to-action,” or “we don’t need to say ‘click here’ in our links anymore…that’s so 1999!”

While many of those assertions may sound snappy and can really catch hold in your marketers’ minds, they aren’t always true. I know a large technology hardware company that had always debated a simple thing like the wording of their main calls to action in their hero banners on the homepage. The visitor would be clicking through to read the tech spec and purchase the item — should we say ‘learn more’ or ‘buy now’? One camp said that ‘buy now’ was too strong for such a high ticket item, and that a softer pitch would attract more potential buyers into the top of the purchase funnel. The other camp said that ‘learn more’ didn’t adequately call the visitor to action, and didn’t exactly make it clear that the visitor could actually purchase the item by clicking there. Simple solution — run a quick a/b test on your next hero banner promotion. Can you guess the winner? In this case the ‘learn more’ call to action increased conversion by almost 50% for those that clicked. I’ve seen it work out the opposite way for other sites, so you can’t just take this as a general rule — you need to test YOUR customers, not just live off of others’ findings. I ran a similar test on my own site, but with ‘more info - buy now’ as a third alternative. It beat both of the other calls-to-action by almost 30%.

Now, the thing to remember is that when you run tests you are not always going to drive uplift with your new ideas. In fact, failing — and failing fast — can be just as valuable as finding a new design that drives uplift. It saves you from rolling out a potential risk to your existing base of business. And between rolling out an untested alternative, and not rolling out any change at all, I think we’d all say that the latter is less risky.

I know one retailer that offered a branded credit card. They knew that if someone applied and was accepted for their store credit card, that customer became very loyal and valuable. Debates had raged internally about whether to offer that credit card much more prominently during the checkout process. There were huge proponents of the card offer being an interstitial offer — effectively the acceptance or rejection of this offer was an extra step in the middle of the checkout flow, to ensure that everyone saw the full pitch. Other big retailers did this hard-sell approach, and some retailers had even attributed their profitability to these credit card programs. When this retailer rolled out the interstitial offer it did increase credit card signups — but not nearly enough to compensate for the 5% drop in conversion, which was statistically significant for their traffic size. They quickly pulled the new checkout flow back and shut off the test. It was a very bad decision averted with minimal adverse impact on revenue.

The tough thing with one of these situations is that once you’ve sat in a room and brainstormed creative ideas for how to improve page X or flow Y, your marketers start getting excited about the new approach. They’re emotionally invested. They’re getting stoked. It’s fresh, and it has all sorts of positive arguments supporting it — saying why it could be the next great thing. Folks have made comparisons to other world-class companies that take this same design approach. The designers take a stab, and it looks gorgeous. Now they love it, because it is “such a superior visual design” — and let’s face it, many designers are Picasso at heart. They love beautiful design, sleek design. And revenue performance is often a 2nd (or 3rd, or 15th) priority. You may even have to involve the developers to build new functionality for the new idea, and they invest time & creative juices into it. Everyone is looking at this thing and thinking it really holds potential. And then you roll out the test, and it doesn’t dominate. That’s a harsh moment. But the thing is, you’ve got to help the team realize that failing fast can sometimes be just as useful as driving uplift. They’ve also got to realize that what worked for another site’s visitors may not be the right thing for your visitors. You have to test to know what’s right for yours, and that’s a discipline that will pay off repeatedly over time.

Tune in next time for another installment of When to Test.

- Brig Graff

However, it is always odd to me when I see a company that has found some very compelling insights in their analytics but they have yet to act on it. They may have even done a study to estimate how much money they could be making if they just improved a particular top entry page’s bounce rate, for example, and it could be worth millions in incremental revenue to them to decrease that bounce rate. But months later, they have yet to realize those gains because they didn’t know where to go from there. As a result it can sometimes seem that they are not yet earning the first dollar of ROI on all that effort and expense to do the analysis. That is, until they take action. And if you never take action on your analysis then it may have been interesting, but largely academic. That is what testing and targeting are for – to help you actually earn your ROI, not just forecast it and document it.

I do this all the time on my own website with the Omniture Suite, and I never cease to be amazed at how simple it can be to take action that drives real results fast. In my years at Omniture, and when I was an Omniture customer, I’ve led many initiatives involving analytics, testing, and targeting. So from this experience, I wanted to share the first in my multi-part series called “When to Test and When to Target.” We’ll start with the first item in The 5 Times When You Want to Test.

1. Landing Page Optimization – when you need to optimize beyond the click.

For many years companies have been spending a lot of money on advertising to drive visitors to their websites, but comparatively under-investing to convert that traffic once it arrives at the site.

I know a large business that as recently as last year was spending almost $100MM of their advertising budget in PPC search campaigns. It’s fair to say that they were only spending maybe 0.2% of that to optimize the keywords’ landing page experiences after the clickthrough. I don’t want to make it sound like it has to be a 50/50 split between ad spend and optimization efforts – that’s probably unrealistic. But $500/$2 ratio is quite lopsided toward the ‘spend side’ regardless of who you speak to.

One quote that has always stuck with me comes from a Wunderman executive who “gets it.” He said the following:

“We preach that clients today should be setting aside 15% of their spend for optimization. We believe that the opportunity to improve the performance of the other 85% is that significant, and well worth it.” – Mark Taylor, Wunderman, EVP and CIO

This means that if you have an advertising budget of $500k for a particular “new product release” campaign, you normally might engage your agency to build a flash microsite to capture leads, as well as a couple of different sizes of display banners to put out on publisher sites. Then point them all to the microsite. But if you are looking to maximize the number of leads you get out of this new campaign, you might do the following instead: Ask your agency to use their creative minds to come up with 3 vastly different approaches to the microsite, depending on differing theories they have of what might drive conversion. Then have them do multiple alternative creative approaches to the display ads. Spend the first 15% of your budget to run some tests to find out which combination of ad+microsite is the best performer. It’s the Darwinian concept of letting the first visitors succeed and fail to discover what works and what doesn’t. Because who’s to say that the first microsite idea the agency came up with is going to be the highest-converting idea? Once you know the best performer, push the rest of your ad spend to that experience. At least now you’ve found the least-leaky bucket before turning the water on full volume.

Returning to the business that spends $100MM on paid search…. The fact of the matter is, in today’s economic climate the VP of Advertising may have his acquisition budget hacked in half. He can’t spend nearly as much just to shove traffic at the site. But, of course, the VP of the .com is probably still held to her same revenue targets from last year. The only logical way for the CMO to make this work is to squeeze much more juice out of the traffic that DOES come to the site. And the way to do that is to constantly be running optimization campaigns on your landing pages. Some of you may be thinking, “That’s great stuff…I wish we could achieve it. But we’re just trying to survive this year.” That’s true – and landing page optimization is your marketing department’s life raft.

Tune in next time when we’ll discuss the second situation when you should test….

-Brig Graff