So, you’ve set up an A/B or Multivariate test. You decide to measure a micro-conversion like click through rate.  The page you’re testing is designed to get users to make a choice – to click this button or that button.  Good design.  Nice test.  Go for it!

And then you see something like this:

It looks like the test variations are going south, tanking, showing an #epicfail. Meh. How can this be?  The new designs were awesome.  Everyone loved them.  The designs addressed the issues and concerns outlined in the user testing.  The data clearly showed that users didn’t like the page.  The bounce rate was too high and the conversion rate was too low.

Ah ha!  There is the clue.  When you set up any test, you use testing software (lets assume it’s Google Website Optimiser (GWO) for now) to measure a chosen metric and the test software does a great job of doing this.  Thing is, there are other metrics that will be affected by the test.  If you measure click through rate on a page, your test designs will likely affect conversion rate and bounce rate among other metrics. GWO doesn’t measure the changes in these other metrics.   Best practice dictates that you couple Google Analytics (or similar web analytics tool) to your test so you can measure all your metrics on a ‘per test variation‘ basis.  This then lets you look beyond the primary test metric and helps you fully understand the economic value that each test variation offers.

Taking the example above (this is a real scenario we experienced recently), the original page exhibited a higher CTR than the test variations.  This was with good reason.  Users were confused as to which option they should click on.  So they clicked, stayed confused and clicked back to click again.  Users clicking on test variations were less confused and so clicked less…but then converted more!

We looked at the goal conversion rate in Google Analytics for the original and test variations and saw the following results:

Double ah ha!  #epicwin!  This confirmed the hypothesis regarding the user behaviour.  Despite the primary test metric failing, the overall economic value was confirmed.

Lessons to learn from this case study

• Always integrate Google Analytics (or other web analytics system) when testing.
• Measure micro-conversions in the test software
• Measure macro-conversions in web analytics

It would have been so easy to flag this test result as a failure when analysing the test result in isolation.  Taking other metrics into account enabled the correct test winner to be chosen.  Which is the correct test winner?  Well, do you notice how the second variation in the test results was the best converting variation? If you analyse test results according to your original hypothesis, this will help you think about more than one metric and to consider economic value: which test variation delivered the best economic value?  This might differ from the primary test metric but the aim of testing is to maximise economic value in the end.