How to Test and Improve Your Website If Your Traffic is Too Low for A/B Testing
Last updated by Rich Page | 13 Comments
One of the questions I receive most from online businesses is how to test improving their website if their traffic is too low for A/B testing. Which is very important because running A/B tests using a tool like Visual Website Optimizer is one of the quickest most effective ways of increasing your website sales, leads and customers!
So to help these websites with low traffic (like small business websites), I’ve created a guide revealing how much traffic you need and techniques for setting up low traffic tests.
Note that this guide presumes you know what pages and page elements are best for testing, which is another topic entirely…
First, check if your traffic is too low to run an A/B test
Before we discuss the ways to run A/B tests if you don’t have enough traffic, it’s obviously important to first understand if you have enough traffic!
How much traffic do you need to run an A/B test on a web page?
To put it in very simple terms, you need at least 1,000 unique visitors per week to the page you want to run an A/B test on. If you don’t have this much traffic, it will take a long time (3 weeks or more, if ever) for your testing tool to gather enough data to determine a statistically significant test result (a great guide on this here). And the more page variations you test (A/B/C/n), the more traffic and time you will need to get a result. But you need something else too…
Is it just about having enough website traffic to run an A/B test?
In a nutshell, no. Even more importantly, you also need enough ‘conversions’ on your website to run an A/B test. This is because to run a test, you need to tell the A/B testing tool what determines success, and this is usually a major goal like a purchase, a sign-up or a form completion. And the less conversions your website gets per week, the longer it will take the testing tool to find a winning result. As a guideline, your website needs at least 50 conversions per week for a simple A/B test (25 per test version).
What happens if I have enough traffic but not enough conversions?
If you have enough traffic to run an A/B test, but not enough conversions like sales or signups, then consider changing the conversion to be something simpler that happens more often, like a click to a specific next page in the purchase/signup flow to determine success. Even simpler, in Visual Website Optimizer you can use ‘engagement’ as the goal, which is defined as a click on anything on the page you are testing.
How much traffic would I need to use a multivariate test (MVT)?
A multivariate is an advanced A/B test, where your tool simultaneously tests multiple versions of different page elements (rather than simply testing one page versus another in an A/B test). This requires much greater levels of traffic, because the tool needs to show each combination of elements being tested to enough visitors to be able to build statistical significance. If you have low traffic, you simply won’t be able to run an MVT.
Try these techniques if you don’t have enough traffic
Don’t have enough traffic? Don’t give up hope about running tests to improve your website! You can still use some techniques to increase the chances of you running an A/B test, or use some alternative approaches to give you similar insights and results. Here are some techniques to try:
a) Buy extra traffic for a short while to the page you want to test. If you can afford it, the simplest technique is to just spend more on driving traffic to the page you want to test. This way you hopefully get enough traffic to run an A/B test, and could driven by using Google Adwords or Facebook Advertising (which I think is better and cheaper). If you have a large enough email subscriber base, you could also try sending an email campaign that drives traffic to the page you want to test. Pretty simple right?
b) Use Google Adwords to test and find high-converting wording. With this technique, instead of creating an A/B test in a testing tool, you use Google Adwords split testing function to find your highest converting headlines and wording that relate to the page you want to test. Here are a few variations to illustrate this:
To do this, simply create a few different ads in Google Adwords that emphasize different headlines and descriptions, turn on the ad rotation function (for fairest results, use ‘rotate evenly’ option), and see which ad version gets the most clicks to your website. Then once you have found the winning ad, replicate that winning text on the page you wanted to test, and watch your conversions and sales grow! Here is a great guide that shows a company that used this with great results.
c) Use a landing page tool to test and find high-converting wording, imagery and call-to-actions. As an even better option than using Google Adwords to run tests to improve your website, you can use a landing page tool that offers basic built-in testing functionality (if you can afford it).
To do this with a landing page tool, create a few simple pages that have different wording, imagery and call-to-actions relating to the page you wanted to test, and find out which version engages your visitors the most. You then simply take the winning content and add that to the page you wanted to initially test. You can use a landing page tool like Unbounce or Leadpages to run this type of test (note, they don’t gather or show you statistical data, so the results won’t be as good or accurate as using an A/B testing tool).
d) Do qualitative testing instead by getting indepth feedback from testers. Qualitative testing is based on gaining more indepth ‘quality’ feedback as opposed to ‘quantitative’ testing that is based on cold, hard numbers in a testing tool.
To do this, create another version of the page you want to test, and then use a tool like Usertesting.com show both versions to testers (you can even use your own website visitors) and ask them to complete tasks based on which page version they prefer and engages them better. Then simply launch the page version that has the best results. This technique gives such great feedback and insights, that I suggest using this in addition to using an A/B testing tool.
e) Use a multi-armed bandit testing tool instead – but be wary. There are A/B testing tools like Myna that claim they can get statistically significant test results even on low traffic websites. How? Well, their testing approach simply needs a much smaller sample size to run the test, with 90% of the rest of the visitors seeing the A/B test variation that is currently winning.
However, I think this approach is flawed and can give invalid results because it finds winning versions much too quickly that might not actually the best ones (doesn’t take into account traffic variances or waiting for fluctuations to settle down). Many experts agree too, so use tools like this at your own risk. And the crazy thing? Even Google Content Experiments uses this approach for its testing!
f) As a last resort do a manual test, but realize the drawbacks. This manual technique is where you simply launch the new page version you want to test for one week, check key metrics at the end of the week, then revert back to current page (the control page) for a week and gather data. After the 2 weeks, you see which version converted the most visitors. This has several drawbacks though – it does not give you statistical significance, and your visitors will see two different versions over the two weeks which can be confusing for them (with an A/B testing tool that wouldn’t happen because it cookies visitors to ensure consistent experience).
Now you know some techniques to test improving your website if you have low traffic! Don’t forget that before running any A/B test on your website, its essential you know best practices for using A/B testing tools and knowing what to test for best results.
Now over to you, have you tried any of these techniques for running tests on low traffic websites? Or which ones of these will you try first?