Google Content Experiments – testing templates and sitewide content
So, from our recent series of posts about Google Analytics Content Experiments, you’ll have seen the level of sophistication of the tests getting steadily more. We’ve tackled simple experiments with vanilla, out of the box tests, we’ve looked at tests where virtual pageviews are used in the Google Analytics tracking script.
Now we’re going to look at tests where you can test content changes across a whole site – not just on one page. Template testing or MVT testing in an A/B framework.
How do you MVT test in Content Experiments?
Before I start – it’s rough and it’s limited. Consider this a work in progress…but it’s a start if nothing else – input and constructive feedback is welcomed!
So, caveat over, on with the show:
The control script needs to change – remove:
And put this in it’s place (or something similar as taste allows but still functions appropriately):
var combination = utmx('combination')
So, we let CE do it’s stuff and set cookies but don’t let it redirect. You can then use the combination variable as you see fit to change content. Put the control script on every page.
At this point, your experiment can register conversions but not pageviews. Notice when you go to a standard experiment variation or control page you see the URL change to include a querystring parameter like this: ?utm_expid=11768512-6
Notice also, the __utm.gif request also contains a parameter: utmxkey=11768512-6 Without the utm_expid on the querystring, utmxkey doesn’t happen.
So, how do we get utm_expid on the querystring without redirecting? Before the GATC, we molest the browser history:
history.replaceState('data', 'title', document.URL + '?utm_expid=11768512-6');
This will affect the URL before ga.js does it’s experiment tracking magic. See this little video of a totally contrived example.
Try examining the __utm.gif request of a CE test with and without the replaceState call to see the tracking in action. Once you’ve done this you can execute the standard GATC, a virtual pageview or whatever. Essentially, you’re running the same test on each pageview – across the entire site. You could just run the test on category or product pages as you see fit.
The limitation – still only 5 variations and I’ve not released this technique into the wild on a large scale experiment so if you copy this technique, tread carefully but good luck!
UPDATE (28 August 2012):
We’ve refined the technique a little. IE (and some less modern browsers) can’t handle the history.replaceState so we’ve tweaked the script to use window.location = blah blah….+ ‘?utm_expid=yadd-yadda’ for the rubbish browsers. Less clean but still useable as a test as smarter browsers still use replaceState – no redirection!
Also, the replaceState functionality needs an extra bit of logic to only fire when utm_expid is not present or you end up with a lot of utm_expid params and it goes wrong.
One last tweak: not only do we rip out utmx(‘url’,’A/B’) but we also remove if(l.search.indexOf(‘utm_expid=’+k)>0)return;
UPDATE (19 September 2012):
We found that when running a test on a subdomain having previously tested on another subdomain (think test.domain.com and www.domain.com or shop.domain.com) the variations being shown were inconsistent. This means that on each page refresh we saw a new variation being presented – not good testing!
The reason for this is due to how the __utmx cookie was being extracted from document.cookie and passed on to google-analytics.com Expect an official update on this issue in due course but in the mean time we’re successfully running this modified control script on our content experiments (modified script in bold):