As I was fairly familiar with the basic product features already, Google’s Tom Leung and I had the chance to talk informally about some of the benefits of putting this product in many hands.
One issue I raised was how to weigh “advice sessions” and “clinics” and the like. The analogy might be a bit along the lines of American Idol… it would be very entertaining to see someone donning a Simon Cowell wig and blurting: “That’s rubbish! That page will never convert! I mean just look at how small that search box is, and the abominable use of tables. And that paragraph about shipping. So trite. In short, I got nothing out of this and I’m wondering right now why I even bother to sit through this.” So in short, Simon’s usability advice could be hit-or-miss.
Compare that with the flipside: a distinctly un-Cowell-like Talent Optimizer that would input various pitches, intonations, arm lengths, dance gestures, wardrobe elements, and facial expressions into a virtual performer… and measure the correlations of each element to positive responses from the paying audience. (“Dr. Clark, it appears the optimal arm length for Celine Dion is a full foot shorter than we’ve been using! Egads! And look at that fingernail data! Midnight blue is kicking butt!”)
Everything in its place. Just as we don’t really quite want a Talent Optimizer (though record labels and boy band promoters probably have something close to that in the underground lab) judging American Idol, we need to move beyond Cowell-like subjectivity in our ecommerce efforts. A multivariate testing process is just that.
I managed to get a lengthy riff out of Leung on the reason Google doesn’t recommend Taguchi optimization. That was something I noticed right away in the Website Optimizer literature, but Leung provided more color on it. The upshot is you really do need to test all potential combinations rather than a reduced combination set, because Taguchi makes poor assumptions around variable interactions. You can semi-Taguchify your process by hand. I won’t bore you with the details but rest assured that whether you go with zero Taguchi, semi-Taguchi, or Taguchi on a Taco, this will likely be a step ahead of a simple A/B test, and many steps ahead of not testing at all.
Because testing is almost always better than not testing – and usually so much better that it even compensates for the risk of “messing with” a home page that ranks well organically – it’s hard to see a significant downside.
Tom and I scratched our heads a bit trying to come up with an answer for the question: roughly speaking, can you make any serious errors running such tests? Setting aside technical snafus and things you might do to ruin your website by misinstalling code (your problem), basically the answer would be no. The biggest “error” would be to pick the wrong things to test – in other words, not improving as much as you could if you did it better.
The maximum number of variations the Optimizer product will allow for a single landing page test? Leung said 10,000. I recently completed a test that involved 16, and am running one with 24 now. For these particular tests, had I done them using 10,000 combinations, they’d reach statistical significance around the year 2258. Coincidentally, that’s about the time they’ll have finally perfected Celine Dion.
