#1 AB Testing Mistake Made by Testing Newbies
A client recently asked me what I thought was the most common issue we come up against when it comes to running AB and MVT tests. This was an easy one to answer as it comes up time and time again. My response was this:
“Clients changing the page that we’re running a test on!” (note this was said calmly and not SHOUTED!)
Changing the underlying page where an AB or multivariate test (MVT) is planned can affect things differently depending on when the change takes place.
If the page changes part way through planning the test, the impact should be minimal. It will require an update to the test plan and perhaps a reworking of the structure (for example test array) of the test.
However when a test has been planned, designed and is ready to be built by a developer the test page should be in a state of lock down (code freeze) up until the time the test has completed and is switched off. The reasons for this are as follows.
If the underlying test page changes during build then at best the developer will need to check all of their work to ensure it’s compatible with the new code base. At worst it could result in the test no longer being feasible. Often the impact is somewhere in between the two – essentially resulting in extra effort and therefore delays.
If the test page changes once the build is complete and it’s either in quality assurance (QA) or has passed QA then the test will of course need to be re-QA’d as it’s highly likely that the test code will no longer be fully compatible with the new underlying page code. There might be elements on the page that the test code is looking for which have now been removed, moved or renamed. If the test fails QA then it will need to go back into build and then passed through QA again. Again, the net impact is essentially extra effort and delays to launch.
Now things get interesting. If the test is live and the underlying test page changes (because a new release is pushed out for example) there are much larger risks to consider. As previously mentioned the test my well no longer render correctly which could result in strange page layouts or even a loss of functionality – for example no longer being able to accept payment from your customers! In addition to this the test will need to be switched off and a decision made as to whether the test will be rebuilt to be compatible with the new code or simply abandoned/cancelled. Needless to say that as the test was still running it’s unlikely that statistical significance would have been reached and therefore the data collected will essentially be useless.
With the introduction of different ways of working (lean, agile etc.) it’s likely that release cycles will be much shorter and more frequent than they used to be. With this in mind it’s more important than ever to communicate with any teams that have the potential to update pages that are being considered as testing candidates for MVTs and AB tests.
If there’s a chance that one area of the test page might change but it’s completely separate from the area of the page which will be tested on then this could be OK as long as it’s documented and planned for correctly. Involve the developer at the earliest opportunity to check with them if this will impact their code.
Most of the above comes down to one key point – communication! Keep talking to stakeholders, colleagues, related teams etc. and let them know what you’re planning. Make sure you’re aware of any changes that might coincide with key test phases (build, QA and live phases).
In short, if at all possible DON’T CHANGE THE PAGE!
Get an invitation to join my private list where I’ll share my CRO experience & answer your questions
Author: Phil Williams
Phil is the founder of CRO Converts. He has had the opportuntity of creating successful testing and personalisation strategies for many of the UK and Europe’s leading brands.