Why You Shouldn’t Compare Optimisation Data with Analytics Data
One question which comes up time and time again is “why don’t my analytics numbers match my optimisation numbers”?
I’m going to keep this short and just highlight the main things you should be aware of when making comparisons between different data systems.
1. Cookie lengths may vary considerably
Analytics platforms will typically use much shorter cookie lengths (30 minutes) compared to optimisation platforms (90 days). That means that if someone returns to our test page once per day over a 14 day period the analytics tool will count this as 14 unique views whereas the optimisation platform will recognise that it’s the same person and will only trigger 1 unique view.
2. Uniques vs. Non-Uniques?
If you’re going to compare data between two different data systems be sure to at least compare unique views / visits / visitors with the same metric in the other system. If you’re comparing unique data with non unique data in another from a different platform it’s highly unlikely the data will even come close.
3. Views / Visits / Visitors?
Similar to point 2 above – make sure you’re clear on the definitions of the data you’re viewing. Are you looking at visits or visitors in your analytics tool? Or is it simply views?
4. What’s being excluded / included?
Oftentimes a test that is being run using an optimisation tool will have a list of exclusions that aren’t reflected in the analytics profile. For example for an ABn test or MVT (multivariate test) we might be excluding the following:
Internet explorer 6, 7 and 8
Automated tracking tools such as Gomez, Keynote etc.
Again, we will see very different numbers if one data set includes the above and the other excludes them.
5. Tracking Differences
The way in which we actually track things will vary across different systems too. Are we tracking clicks using an onclick conversion? Or are we tracking when the user loads the next page? Does the system use a log file system or a beacon system? What are the different timings used in the data systems? Is there a time out associated with the test for customers on slower connections? Are these people being counted in analytics but not in the optimisation data?
Our first response to seeing numbers that don’t match is that something must be wrong with one of the data sets. Actually, variance between systems is expected. According to one trusted source the variance between different data systems is usually between 35% and 50%.
We can save a lot of time by looking at the data we’ve collected and asking ourselves if the conversion rates and views that we’re seeing are consistent with what we’re expecting. If they are then our time would be better spent analysing the data we’ve captured to predict which variants are comparing best rather than trying to make one data set match another.
Author: Phil Williams
Phil is the founder of CRO Converts. He has had the opportuntity of creating successful testing and personalisation strategies for many of the UK and Europe’s leading brands.