How to Run Accurate A/B Split Testing

Imagine you have a reasonably popular website that receives 10,000 visitors per month and has a conversion rate of 3%. As a data driven organisation, someone decides that it would be a good idea to run a test to see if changing the colour of a button on the site has an affect on the conversion rate.

The graphic is changed and three days later, the average conversion rate of the website has gone from 3.0 to 3.2%. Is this significant? Nope, statistically speaking this is just noise.

A Bernoulli trial is an experiment that governs the statistical accuracy of a two-tail test. Moving from 3.0% to 3.2% conversion rate because a button changed colour requires that each of the two samples (the old colour versus the new colour) has to have a sample size of 57,697 if we are to be 95% sure our decision to change the button is the correct one.

That means we need to have 115,394 people partake in our experiment before we can say with relative certainty that our button colour change decision is correct. With this website that means the test needs to run in isolation for over a year! Even if we drop the accuracy of the test to 60% certainty (remember chance alone gives us a 50% certainty)we are still talking about a two-month trial. This is impossibly long.

The numbers change drastically depending on the size of the swing, the accuracy we are prepared to accept and the duration or sample size we tolerate.

So should low traffic websites avoid running A/B tests? Absolutely not, but for those with highly trafficked sites which tests you choose to run becomes super important. When we read about how Obama optimized his campaign website attributing to a 49% increase in donations, what we are not told is the sheer volume of traffic the site enjoyed to run its tests effectively.

Changing button colour rarely has the dramatic effect we often read about. Choose your tests wisely. Bigger swings in conversions require less testing than minor changes. There are loads of web calculators to help with the binomial mathematics when preparing your test, just Google “Bernoulli test calculator”.

If you would like to talk about your online channel conflict challenges, why not phone Niall or Edward in iON on +442890455911 or drop them a message via their website

Follow the author

Follow iON

About The Author

Niall McKeown is the CEO of iON and author of the forthcoming book 'The 7 Principles of Digital Business Strategy'.