Reflections on the Last 7 Years of CRO

Reflections on CRO over last 7 years - Road picture

I’ve been in conversion rate optimisation (CRO) for about 7 years and a lot has changed in that time. The year 2010 was when Optimizely was founded. You probably weren’t using Google Chrome to read this blog. Most of us hadn’t even heard of Chrome back then.

When I first started in CRO, it was all about button tests and MVTsResponsive design was pretty much unheard of. Most people didn’t know what conversion rate optimisation was and even if they thought they did, as soon as you mentioned the word ‘optimisation’ they’d start talking about SEO. So what has changed?


Digital Marketing has Embraced CRO

7 years ago CRO wasn’t really recognised as a major player in the digital marketing world. SEO and PPC were well established but still the more recent additions to a digital marketing team. It was our job to educate and convince anyone who worked in a digital marketing team that they needed to start testing.

Hiring was a challenge too. Trying to find an ‘optimisation analyst’ or ‘optimisation consultant’ was next to impossible as very few people had this experience. Instead, we had to look for skills which would be transferable and hire people who could learn quickly and were analytical by nature.


Companies have Adopted CRO

In 2017 most large organisations have at least one dedicated CRO resource (am I the only one who hates that word?). Their job title may be ‘optimisation consultant’ or ‘optimisation analyst’ or something more generic like ‘digital marketing manager’ but the majority of their time is dedicated to CRO projects. Some of the larger companies I’ve worked with have several people who are dedicated to CRO projects. A few years ago this certainly wasn’t the case. Instead we would be liaising with people in digital marketing teams or perhaps analytics teams who had very minimal understanding of conversion optimisation. The fact that companies are putting their money where there mouth is shows how seriously they value this digital marketing discipline.


We’re Running Less MVTs

This may not be completely across the board but I’m definitely seeing a lot less demand for multivariate tests (MVTs) these days. I think part of it may be that its simply gone out of fashion a little. But there’s more to it than that. Those doing CRO are getting better at using analytics data to fuel their test hypotheses when planning tests.

In the past, especially for high traffic sites, there would be a desire to analyse all aspects of a webpage in one hit. We’d therefore use an MVT (with fractional factorial methodology) to try to prove exactly which aspects of the page were working and which weren’t. The trouble with that approach is that you’ll end up having to run your tests for longer periods of time in order to get any results. You’ll also spend more time planning, building and QA’ing the test. With an AB test you’ll likely have a much shorter time span between planning the test and getting your results.


Test Complexity Has Increased

This may sound contrary to the previous point but ironically the MVTs we used to run were often easier to build and QA than many of the AB tests we now work on. For example, a popular MVT consisted of testing button colour, button size and button copy. If we tested 3 versions of each of those we would end up with a 9 experiment test (using a fractional factorial 3-3-3 array).  Despite the fact that we’d need to build 9 experiments it’s actually quite easy to code those changes.

AB tests in 2017 often span multiple pages, on responsive sites across highly dynamic pages. This is a web developer’s worst nightmare. Creating a test like this requires serious planning up front by your CRO consultant. All design and functionality aspects need to be considered. It may mean we only run the test on certain breakpoints and we may have to simplify some aspects but either way there’s a lot more to think about now than when we could just run a good old fashioned button test.


Conversion Tracking Improvements

Something else that has evolved is the way we track our tests. In the past test tracking was relatively simple. We’d track people loading a page URL (e.g. confirmation page) or clicking on a particular element on a page (e.g. a button). While those goals are still valid and feature in most test plans I come across we are now becoming more sophisticated with our tracking requirements.

For example, perhaps we don’t just want to track if people clicked on a form button. What we actually want is to only trigger the conversion IF no validation errors have been thrown. Or perhaps we only want to trigger the conversion IF a visitor has clicked on a particular link earlier on in their journey. Implementing these sorts of conversions isn’t complex but does require clear communication between the consultant and the developer.


Segmentation and Personalisation

Having the option to segment and personalise an experience for visitors isn’t new. The difference is really just the frequency in which we do this now. A few years ago it was fairly unusual to try to segment a test beyond the basic desktop/tablet/mobile segments. In recent years it’s become the norm. We’re not just looking at out of the box segments such as geolocation, browser type and time of day. We’re now doing the more fun stuff. A recent retail example was feeding in a segment of users who we knew had previously browsed different jean types (skinny, bootleg, boyfriend etc.). We then used this data to personalise their experience when they return to the site. We tested the impact of re-targeting based on previous browsing behaviour.


Less HiPPOs

I’m not sure this is necessarily proven but it certainly feels like there are less decisions being made by HiPPOs.

CRO Changes - Hippos

Instead there is a definite move for companies to be “data driven“. This is is clearly a good thing for those of us in the business of proving hypotheses using data. The prevalence of CRO has empowered marketers to challenge their leaders to ‘test and prove’ before changes are made.


More “CRO Agencies”

With the rise of the CRO has come the spread of the CRO agency. Often this means that a digital marketing agency has spotted an opportunity to make more money from their existing clients. They claim to now offer ‘CRO services’ but in reality have only a surface level understanding of conversion optimisation. Perhaps that’s just my skeptical view. There are of course very legitimate agencies and tools out there – a lot more in fact than 7 years ago.



A lot has changed in CRO over the last 7 years and I’ve learnt loads. The popularity of CRO has spread and tests have become more sophisticated. I’m looking forward to seeing what the next few years will bring. In such a fast paced world there will no doubt be many exciting developments that we can’t yet imagine. One thing is certain, CRO will continue to bring business growth to those who adopt it.


Published on May 15th, 2017

Author: Phil Williams

Phil is the founder of CRO Converts. He has had the opportuntity of creating successful testing and personalisation strategies for many of the UK and Europe’s leading brands.

Share on FacebookShare on Google+Tweet about this on TwitterShare on LinkedIn
No Comments

Post A Comment