We caught up with Michael Rippon, Managing Director at Raindrop Digital who discusses the importance of A/B testing, its benefits and outlines how A/B testing affects your SEO.
You’ve looked at your site and found areas that aren’t working.
It may be that your users struggle to use a certain area of your site, your messaging around CTAs isn’t clear, or your website just needs general improvement (remember, sites can always be improved).
But now you have multiple different ideas on how to solve the problem.
How do you choose which one to go with?
A/B testing.
What is A/B Testing?
You may have come across split tests or bucket tests, which are just other names for A/B tests. Essentially, they are all methods of comparing two versions of your website, or a specific web page against each other to see which one works the best for you and, more importantly, your users.
A/B tests work by randomly splitting your audience as they visit your website to show version A to one half and version B to the other (hence the name).
A/B testing is a form of experiment that is based around a conversion goal on your website. Like with any experiment it can get a bit complicated to understand as you’ve got to get your head around statistical analysis (flashbacks to A-level maths).
But we can simplify it for you in our step-by-step guide below, giving you a checklist to follow whilst conducting your own experiments.
A/B Testing to Success
Testing removes the need for guesswork in website optimisation and means that your decisions will be data-driven (just what us at Raindrop like to recommend). There should be no thinking in CRO, decisions that start with “we know” are far more reliable. And that knowledge is exactly what A/B testing can give you. You won’t just think rearranging your navigation will make it easier for your customers to use, you will know it will.
Seen as A/B tests are purely based on your users, of course, they also bring the benefit of improving user experience. An improved user experience will mean more happy customers that are more likely to return. Winning.
But before you get carried away, is an A/B test a feasible option for your business?
Sample Size for A/B Tests
As mentioned briefly above, A/B tests rely on statistical analysis so there does need to be enough traffic coming to the site or app to be able to get statistical significance.
To determine how much traffic is required to be able to A/B test we recommend using one of the following calculators:
- https://vwo.com/tools/ab-test-duration-calculator/
- https://abtestguide.com/abtestsize/
- https://www.evanmiller.org/ab-testing/sample-size.html
There are many more out there, but these are a great start. By playing around with the detectable effect, you’ll be able to see what kind of change you need to make with the traffic you have to claim statistical significance uplift.
The normal rule of thumb is, a large big bang change will need less traffic than something like a copy change. This is because a big bang chance should massively change user behaviour, so you will be able to see a clear difference. Whilst a small change might only move the needle by 0.01 of a percent, so you need more traffic to be sure of this.
The biggest rewards normally come from resolving issues further down the funnel such as a shopping cart or payment journey. However, these parts of the site normally see far less traffic than the likes of a homepage. So, if you’ve used the calculators from above and a test will take months to run we would recommend starting at the top of your funnel, working your way down, optimising each step and sending more qualified traffic through the funnel.
When should I A/B test?
You might want to A/B test a website, an app or even an email. You could be testing the placement of your call-to-action, the layout of the navigation or a whole redesign of a page. Whatever you need to change, let A/B tests provide you with the data so you know how to change it.
Let’s take the hero section of your site’s home page.
You can see from a combination of Google Analytics, heatmaps and video recordings that the two CTA’s at the top of your homepage aren’t getting clicked on, and more importantly, aren’t turning into sales or leads. This is great, you’ve already based your decision to make a change to your site on data.
But how do you solve this identified problem?
The CTA links are currently at the top right of the page, one on top of the other, in black text.
Do you:
- Move the CTAs in to the middle of the hero section side by side?
- Increase the size and change the colour of them to give more prominence?
- Change the messaging around them to give more context?
You’ve come to the conclusion that you need to change the site based on data, so make your decision of how to change it based on data.
With A/B testing, you can trial two versions of your updated hero directly to your users. You can see which version performs the best and leads to the most conversions in a given space of time. Then, you will know how to change your hero and you can roll it out onto the live site.
A/B testing is the way forward
A/B testing can do so much more than just settle a one-off disagreement, it can be used consistently to continually increase your goal completions and consequently conversion rate over time. As we’ve said before conversion rate optimisation isn’t just a one-off process, it is about continuous improvement.
Forming your hypothesis becomes easy as just looking in your available data and seeing which changes need to be prioritised in relation to helping you achieve your overall end goal. You should assign resources to the most critical elements that will make the biggest changes towards achieving your business goals.
By focusing your business processes around carrying out A/B tests before implementing changes to your site, app or email, your team will start to build an optimisation culture. This culture will mean that your whole team is focused on continuous but incremental improvements that will in turn lead to continuous growth for your business.
One challenge, however, is building a testing culture with limited resources. If you’ve already got some of that optimisation culture in you, however, I don’t need to tell you that you should research all tools carefully to decide which to use. But there’s a reminder; take advantage of free versions until you get to a point where you need the full package. Which won’t be long if you continue to grow.
A step-by-step guide to A/B Testing
A/B testing is an iterative process that can be represented as a cycle, beginning and ending with research. If you’re constantly collecting data, you’ll have a constant backlog of things to change, tweak and test on your site. This way, you don’t even need to decide what to change and in what order, your data will tell you this.
1. Define experiment
First, determine what element you need to improve. As mentioned above, your analytical data can signify where you should begin optimising based on your current users’ behaviour.
You may find there are a number of variables you want to test; however, it is important that you only test one at a time. Test multiple and you risk not knowing which was actually responsible for any changes in performance.
Based on your data, create a hypothesis for the change you want to make. These should be goal oriented; an example of a goal could be a purchase, an email sign-up or a button click. Your hypothesis should outline why the change needs to occur, what actions need to be taken to carry out the test, and the projected result if your hypothesis is correct.
In defining the experiment, you should also think about:
- Success metrics
- Audience and traffic allocation
- Sample size required
- Test length required
2. Create variation
Identify variations to solve the problem using A/B testing software (we use Optimizely). With most testing tools you get the option of a code only or visual editor. Both can be used to setup and load in external JavaScript and CSS files for the test.
Taking our CTAs in the hero section example from earlier, this could be:
- Version A – The CTA buttons in the middle of the hero side by side
- Version B – The CTA buttons made larger and a more prominent colour
There are three types of tests that you can run depending on what changes you are making and how nifty your coding skills are.
- A/B/n tests are more developer focused, involving the recoding of JavaScript and CSS to create new components and functionality.
- Multivariate Tests (MVT) can be just as simple or complex as an AB test however in an MVT you test multiple combinations of different components. So if you had a new Hero and a New Footer you would have test 4 variations including control.
1. Hero alone
2. Footer alone
3. Hero and Footer together
4. ControlFor an MVT you need a lot more traffic, this is often why we recommend testing one hypothesis and element at a time.
- Split URL tests are suited for full redesigns or to test different landing pages from ads and they are often easier to make in certain CMS systems. In this case, you will end up with example.com/a.html and example.com/b.html.
Depending on the tool you implement for testing, you’ll also have the ability to set up goals to track the experiment. We won’t go into the details of this now, as each tool is different, but the main thing to keep in mind is that you need a primary goal as your main success factor. This should align to your hypothesis with the view to reject or accept your hypothesis depending on the outcome of the test.
Most tools have updated their approach to ensure they account for false discovery rates; however this is something you need to be aware of.
Just before you run the test, of course, you need to quality check it. Your test could look great on desktop, but mobile may not. Make sure to check across all devices and browsers for full functionality.
3. Run experiment
Now all that’s left to do is to run the experiment and wait for your users to participate.
You may wish to do a soft launch first to mitigate risk and check the test is tracking correctly. As soon as you’re happy with it, you’re all set for full launch.
Your tool should automatically assign either version A or version B whenever a user clicks on to the page where the test is running. Any interactions with the CTAs will be tracked to see which version performs better.
4. Measure results
Tools such as Optimizely have evolved to use sequential methodologies in their analysis, this means that you should be able to look at the results at any point during the experiment and see a valid result.
We, however, would still recommend that you leave the experiment running until you’ve reached the required amount of time needed in your sample size calculation. If your sample size calculation says you need less than a business cycles worth of data we would highly recommend you round up to a full cycle.
Once you’ve reached the required amount of time you should have a clear indication of whether the variation is winning, losing or the experiment is inconclusive.
5. Implementation insight
If the test failed, there’s no need to panic, research by conversion.com states that over 70% of tests fail. Even if it doesn’t prove your hypothesis, there will still be valuable insights that can be taken from it. And you can just run another test; this is a bonus of A/B testing being low cost.
If the test succeeded and suggested a change to implement, brilliant. You can implement this onto your live site.
But don’t stop there.
You’re now back round at the defining experiment stage so get back in your data and find something else to improve.
Common mistakes in A/B testing
Mistakes in your A/B test can lead to false positive results. These may be:
- Saying something works when it doesn’t due to sample size.
- Saying something doesn’t work when it does.
Some of these we’ve covered throughout, but here’s a quick overview of common mistakes to go back and check you’re not guilty of that could result in an unreliable test.
- Invalid hypothesis
If you’ve just plucked your hypothesis out of thin air, it’s not going to be valid. Hypotheses should be based on data you’ve collected previously to ensure that your results are measurable and can prove your findings to be true.
- Testing too many elements together
How are you going to know which change resulted in a 20% increase in conversions if you’ve tested three changes at once? You aren’t. Test one at a time so you can track them accurately and see which change performed the best.
- Testing for incorrect duration
Tests need to be run for long enough to collect enough data, but not too long to appear as if you’re trying to deceive search engines. We recommend a minimum of 2 weeks with a maximum of 4/5 weeks depending on your business cycle.
- Testing at the wrong time of year.
Seasonality and other variables can affect your results. For example, an ecommerce business is likely to get busier than usual over Christmas, so running a test over the Christmas period wouldn’t give accurate results.
- Using the wrong tool
It is important that you choose your tools wisely when A/B testing to ensure they have the full capabilities required for an accurate test. Don’t fall into the trap of trying out your CMS system’s plug in, look for a purpose-built A/B testing tool.
- Ignoring statistical significance
Waiting until you reach statistical significance is the bane of A/B testing, we know. But stop your test before this and your results become null and void, they won’t prove anything to your manager. So stick it out, it will be worth it.
Will A/B Testing affect my SEO?
A/B tests pose no inherent risk to your website’s search rankings, as Google permits and encourages them.
There is, however, a few best practices to follow to make sure you don’t fall into a trap:
- No Cloaking
Cloaking involves showing search engines different content to what an actual human visitor would see. Some A/B testing softwares swap content on the page using JavaScript, so users see different variations of the page and bots see original content only.
You can avoid this as long as you don’t specifically direct that Googlebot user-agents should see different content to a human visitor. - Use rel=”canonical”
To make sure you don’t confuse crawlers with duplicate content when running A/B tests with multiple URLs, set a rel=”canonical” directive in order to point to the original version of the page. This will help point bots indexing your website to your original page rather than trashing your ranking.
- Use 302 redirects, not 301s
A 302 redirect is only temporary, a 301 redirect is permanent. You’re A/B test won’t run forever, make sure the search engines know to follow the original URL not the test one.302 redirects signal to search engines that, whilst indexing the original page, they should check back periodically to see whether they should index the variation page. These can be used again in future tests when you want that content to be indexed when content goes live.
- Only run experiments for a necessary time period
As briefly mentioned, running tests showing variations of pages for too long can be mistaken as an attempt to deceive search engines. Keep your tests only as long as necessary. We recommend 4/5 weeks maximum.
Now you know why A/B testing is the way forward for your business, get started experimenting towards a better experience for your users.
