AB Testing - Get the most out of the Experiment Function in Analytics

Netmatters Ltd
Posted by Netmatters Ltd
17th July 2015

What is AB testing?

Split testing is the act of directly testing two different web pages against each other to see which one has a higher conversion rate. The way it works is you have a web page and you want to change a certain aspect on that page, for example you want to change the background colour of your page. What you do is you change the colour on A, and leave it the same on B. Half of the people that visit your site will see site A and half of them will see site B.

This is all simple and straight forward methodology, once you have done this you simply allow it to run this way until you have sufficient data, then you compare the data of the 2 pages and see which one achieved a higher conversion rate. Whichever one had a higher conversion rate is the page more effective at making traffic convert into sales or quotes.

In this sense it is a very easy way to improve your website for higher conversion rates. Something to note is that when you do this split testing you must only change one aspect of the page. If you start changing more than one thing then you will not be getting directly linkable results as to what caused the page to change. You could change the background colour and the button layout on your page and get the same results for both, but it could occur that the buttons were very effective and the colour change was not, causing a discrepancy in the results. You must test one thing at a time to get a conclusive result about which factor is actually effecting the conversion rates.

Why should you do A/B Testing?

If you want to have a strong online presence that causes a lot of people to not only visit, but actually buy your products then A/B testing can be very helpful. A/B testing is very cheap, much cheaper than paying for more traffic through advertisements per say, and it might actually get you more sales than paying for an advertisement on a pay per click basis can. You could have a million people click an advert or a link to your site, and if out of those one million people three people buy something then you have essentially wasted the presence of all of that traffic. You could do split testing and get 1000 visitors, then 4 of them buy your product from the page you changed and you have already made more back at a hugely reduced cost.

With A/B testing you can customise your site completely, change every aspect of your site until you find a combination of aspects that work the best for conversion rates, and then do the advertising. You would see a huge spike in sales, as more of those people that visit your site are actually buying a product.

A good method to follow when A/B Testing

When doing A/B testing you should follow a very standard experiment method. Before your testing you should always ask yourself what the actual reason of your test it. A very basic example of this question would be something like why is my bounce rate so much higher than my competitors, or why is my average session time half as long as the market average. These questions set a base for your testing, and in your tests you should be trying to identify the answers behind the question you asked before you started the test.

You should also always do a little background research before starting your test. You need to understand the behaviour of your visitors before you can try and change that behaviour. Use tools like Google’s own analytics, or any other analytics tool you have available. After doing your research you should try to put yourself in your visitor’s shoes, go on your website; see what detracts from the experience. If you can identify a factor that you believe could be part of the issue then make it your hypothesis, a good hypothesis would be something like “shortening the length of the headers will increase the average session time”.

Now calculate the length that you need to run your experiment for, this should be discovered by analysing your traffic and current conversion rates. Many AB testers will be able to do this for you, and the only real thing you need to do here is collect enough data for it to be accurate and show a real decree of change between the base and control. You can find some A/B duration calculators if you need help here.

Here you have to test your hypothesis, the hypothesis you made above should be a metric that can be tested and should aim to fix that hypothesis. Make your test site wide, and ensure you have a control site (site that is not changed) and one that is changed as variation.

After you've done the testing you need to analyse the data to determine if the changes have actually made an effect on your hypothesis or not. If it has then you will see that, for example the bounce rate on the page that you changed for the test should be different. If it has made a difference then it’s a success! You have directly determined one of the factors that affect your bounce rate or conversion rate and so on. If it has not made a notable difference then go back to step 3 and create a new hypothesis and repeat, remember not all A/B tests will be successful so if yours wasn't then try again.

Guides See How We Do It