Let’s say you have two versions of a design and you want to see which design converts users better. You show “Design A” half the users and “Design B” to the other half. Say “Design B” does better. Now you have your answer: use “Design B”.
But let’s say you have three designs. You start out:
A vs. B = B wins
so you pair up the winner vs. C:
B vs. C = C wins
Is C automatically the best because it beat the winner of the first test? Probably. Well. Maybe? To find out, you pit them against each other.
A vs. C = A wins
Uh oh, now all three designs have one victory. Which is best?
Well you’d probably not use win/lose as the criteria, you’d test actual conversion numbers. So you’d run all three designs for equal lengths of time and whichever had the highest numbers is the best of all three. That means you can’t run one test then later do another, you really need to do A/B/C testing so the data is all during the same period so less unknowable external factors are involved. Also that the length of time is long enough that randomness is mitigated.
So yeah, not really “confusing” so much as A/B testing isn’t the right term when comparing more than two things. More like A/B/../X/ testing or “Let’s compare the effectiveness of some designs and not screw it up testing.”
It’s actually called A/B/N testing. ‘N’ being ‘some number’ – or you could call it single-factorial testing. Are you testing a specific section of a site? Or is it the entire layout
Completely different designs (ex. a Landing Page) can yield a quick result which is good for getting buy-in from a stakeholder. Once you’ve sold them that testing is a good thing it starts to require a holistic approach.
In order to keep it from seeming like uncontrollable, roll-of-the-dice, voodoo its good to look at it completely from the customer’s perspective. Dig deep into the analytics, revisit the user personae, etc.
I’m currently running Test & Target (front-end dev, design, and admin) here in Tampa and at first it’d been a constant struggle to quell some of the fears that some had about it threatening the integrity of design. Hey, maybe 3-legged unicorns under double-rainbows are really cool at the moment but that’s probably not the best for brand identity in the long run.
Anyway, back to your test. Was there a winner with at least a 96% confidence rating? Did each experience (or ‘recipe’) have at least 100 conversions each? If not, no real winner for any of the pages as a whole. Do a follow-up test with inclusion/exclusion. Run a test with one element taken out per experience. Might find some elements hurt and some help. Get the results and ask “Why did this win?” get your hypothesis together and measure it in another test. The more testing you do, the more you learn about your customers – so long as it’s not just a bunch of wildly different designs one after another after another.
Its also really important to look at events occurring outside the pages you’re testing. Maybe its an SEM campaign that kicked off and shifted the demographics of the traffic coming in. Or maybe its an event in the news. I was doing freelance testing work for a travel insurance company back when I was living in Tokyo. We were testing the addition of a travel-related news feed on the home page. A few days into the test we see a sudden huge lift in purchases. It was great, everybody was excited, until we saw in the news there had been a tragic plane crash at an airport in China. I never knew a good feeling could leave so quickly. My point is that the world around us affects testing.
If you’re interested, here’s some really great webinars:
http://www.marketingexperiments.com/research-topics/research_archive.html
They’re about an hour long each, but really good.
Let me know if you’d like any more-specific help with a test. Cheers.