You have a great idea for a different page layout, some new copy or image, or a promotion that will give your site the kick in the pants it needs to break through stagnant conversion. You want to make the change, but you have been through this drill before. Some efforts make things better, some of them don’t, and for the rest, it is unclear, so you assume they must have had some positive impact.
This time through, you are going to test the idea against what you already have so that you know that you won’t lose ground. You have decided to enter the ranks of scientific marketers.
In this column, we will clarify the muddle of techniques that can help you more reliably increase the conversion of visitors to customers on line, including A/B testing, split run testing, multivariate testing, and even something called Taguchi testing that sounds suspiciously similar to a “virtual pet” popular in the 90s.
Scientific Optimization
At its heart, scientific optimization uses the computer to increase the odds that the page or session that your customer sees is the best possible. I find it useful to break the approaches into three levels of sophistication:
Level 1: A/B Split Testing – Simple test of one element of a page against another to see which is more effective.
Level 2: Multi Variable Testing or Multivariate Testing – Testing more than one element at a time to test new page treatments or offers
Level 3: Experimental Design – Using advanced statistics to determine the “best” layout or configuration of elements using the smallest possible number of visitors.
A/B Split Testing
Testing two alternatives at the same time is the easiest but often least valuable way to improve conversion through testing.
Most companies begin scientific web site optimization using AB Split or Split Run Testing. AB split testing allows you to randomly divide your visitors into two groups and show each group a different version of a page to determine which version leads to higher conversion, average order value, application completion, or other target. These visitors are then tracked and a report is generated that describes the impact of the A or B page version on this outcome.
A common use of the AB Split Test is to evaluate the impact of a new page layout on the likelihood of generating a sale. Two versions of the page are created. Often companies test a new page design versus the existing design. Traffic is randomly split between the two pages using custom programming, by splitting the application servers or using ASP tools like Offermatica. The visitors are identified as belonging to the A or B test group and are watched through their visit, and occasionally across multiple visits to see if they are more likely to purchase based on which page they saw.
AB Split Testing is a simple approach to letting your customers “vote” on changes through their behavior. It also suffers from significant limitations. First, many changes have either a negative or not measurable effect. Basically not every element on a page influences a purchase, making it necessary to run several tests in a row to find the element that matters.
To add insult to injury, it can often take 10,000 visits and 50-200 orders over a minimum of two weeks to achieve confidence in an outcome, so running enough tests one after the other can take a long time. If a significant event happens during that period, like Valentines day or tax time, it can compromise the result.
The second issue is that an AB Split Test of a new page treatment against the existing may confirm a positive impact, but it cannot tell you what elements of the new design actually made the contribution. Imaging a situation where a new copy treatment on a home page was very effective, but the new navigation was actually worse. The overall test of the page may show a slight improvement, but hides the fact that you could have done even better with just the copy and not the new navigation.
Multiple Variable Testing or Multivariate Testing
Multiple Variable Testing isolates the elements on a page and helps you find out what elements matter and which combination is the strongest.
One way to improve your chances of finding a winner is to test 3 or 4 or more versions instead of just testing old versus new and pick the best performer from among a the larger group. This is called A..n testing and improves your chances of finding a winning version, but also increases your content development burden. A better way is to test elements on the page in different combinations of “recipes.” This approach is called Multiple Variable Testing or Multivariate Testing.
Multi Variable Testing allows you to test the elements on a page that you believe impact sales. When planned and executed carefully, Multiple Variable Testing virtually guarantees a positive change over your existing page and offers insights into how to market to your customers and prospects elsewhere on your site.
A Multi Variable Test on a product landing page might test the product image, the headline and the product description copy. The goal is to create the most compelling page possible so that visitors to this page, often paid for through search or banner advertising, convert to customers at the highest possible rate. Two or more alternatives of the picture, description and headline are created and a page is composed for every combination of these elements in each of their versions. If there are 3 elements with two alternatives, this requires 8 combinations or “recipes.”
By splitting the traffic randomly and showing each visitor only one version, we can determine the optimal recipe. The advantage of Multi Variable Testing over AB Split Testing is that you can nearly always find a recipe that outperforms existing. The problem with Multi Variable Testing is that if you have more than three elements or more than two alternatives, the number of combinations becomes so large that it takes too many visitors to run a conclusive test.
Advanced Testing and Automated Optimization using the Taguchi Method
The Taguchi Method is the most powerful and most likely testing method to create a significant improvement without creating an overwhelming amount of incremental work.
If you have four elements in a multivariate test including the product picture, headline, copy, navigation and a promotion, and you have four alternatives for each, you need to run 64 recipes. It still takes 40-200 conversions for each recipe to achieve a conclusive test and the volume of traffic required is too great for most applications. Because of this limitation, experimental design methods have been created to test a small, indicative subset of recipes and estimate the theoretical best recipe even if it was explicitly tested. This approach is called fractional factorial testing and can be done using a number of methods including the Taguchi Method.
The Taguchi Method was developed 50 years ago and has been used with great success to optimize automobile and other product manufacturing. More recently, The Taguchi Method was applied to direct mail and web applications. The Taguchi Method takes a number of elements on a page with one or more alternatives for each element and dictates exact combinations that will allow you to estimate the positive or negative effect of each element/alternative.
There are three extremely exciting aspects to this approach. First, by creating a “best page” using the best performing alternatives for each element, significant improvement can be achieved. Second, the length of the test cycle and the number of visitors required is surprisingly small. And finally, since the “recipes” are created using modular element/alternatives, using a solution like Offermatica, Taguchi tests can be designed and executed in a surprisingly small amount of time.
Taguchi tests have been run on email, PPC ads and Landing Pages with great success. Where an AB Split Test might create a 5-10% improvement, a Taguchi test cycle will regularly return 25-45% improvement and has been known to improve results by 100% or more. A test cycle includes two weeks of testing a large number of elements in just two alternatives to identify which elements increase the likelihood of converting a visitor to a customer, a second test where the high-impact elements are tested with a greater number of alternatives, and a final test of the “best recipe” against the original page. The test cycle takes from a couple of days to a month depending on traffic and variance and can be designed and run without significant quantitative marketing or statistics experience.
After using Taguchi Testing on a single page to optimize the conversion for general traffic, things get a little more cutting edge and complicated. Early attempts have shown promise in applying the Taguchi approach to multiple pages, the entire session and even multiple sessions. Also tests can be targeted to optimize the page or pages to a subset of visitors.
Scientific website optimization using AB Split Testing, Multivariate Testing and Taguchi is not a substitute for great marketing ideas, or a strong sense for what will work. The raw materials for great test results are always great ideas. But finally, it is possible to quickly and easily quantify these great ideas and see the result on the bottom line.
Testing is not a replacement for marketing, and no technology will design your pages for you. However, these alternatives can provide a reasonable map of how to use technology to make your brilliant ideas hit home.
For more information about AB Split Testing, Multi Variable Testing or Taguchi Testing please contact: mjroche@offermatica.com or visit: http://www.offermatica.com
Matthew Roche has a BA from Yale University and fouded Fort Point Partners Inc. now called Offermatica. Under his guidance, Fort Point build ecommerce applications for Nike, JCrew, Best Buy and many others.
Offermatica has captured the experience gained with these ecommerce leaders and prouced a plug and play ASP testing platform that brings quantitative testing tools to boost online sales.
