December 20, 2016
A/B Testing, or split testing, is a method of comparing two different versions of the same webpage or application against each other to see which one performs best. Alternate variants of a page are shown to users at random, with statistical analysis revealing which one better achieves a certain goal or conversion metric.
Or to put it in Lehman’s terms and from the user’s perspective – A/B Testing is like dating twins. The best part? You get to play the role of cupid! So you set up a blind date between Twin 1 and one of your users. On the date, Twin 1 has a tendency to talk WAY too much and this may spoil the chance of romance. So on date number two, with Twin 2, you introduce a muzzle or gag of some sort. It’s a subtle change and everything else remains the same; but this may improve the chances of your user getting a word in and obtaining romance!
This is pretty much the same principle as A/B testing online. You could have two different versions of a product page – one with the ‘Buy Now’ button at the top of the page and one with the same button placed at the bottom of the page. Over a set period of time you will start to notice one page converting better than the other and therefore this is the one you go with moving forward.
The changes you make between your page A and page B can be small or major; ranging from headline changes to full page design modification. The important bit is the analysis and discovering which version works best. This allows you to know what is working rather than relying on guesswork.
So you have your set of twins and you want to set a variance. For example, this time let’s say the difference is the colour of attire. Twin 1 is wearing red and Twin 2 is wearing blue. The chances of romance and getting to second base will then ultimately depend on which colour appeals to your user more. Everything else is the same, it is only the colour of attire that has changed. But is this enough? You will know which variant works best but will not particularly know why. Why is blue better and red, or vice versa?
This mystery can be solved if your user is allowed some info on the twins beforehand Otherwise, this is a totally blind date. Providing some details upfront will allow your user to better prepare themselves for the date and will provide more insight into why one of your twins is preferred over the other. For example, finding out what your user is looking for in a date will be a massive help to you beforehand. This is where other forms of multi-variate testing come into play.
True Intent Studies intercept visitors to your website or app, requesting them to reveal their intentions for visiting in advance. The study will inform you of who is visiting your site and what their aims are. By capturing this information you can analyse the attitudes and behaviours of your visitors and analyse them against their intentions. The results of these studies will empower you to make informed decisions on pain points and what should be tested. Going back to our dating analogy – this is perfect! This is like finding out what your user’s intentions in life are, and why they may be interested in your twins, before the date itself takes place. Imagine!
“Your user’s name is Tracey, she wants marriage and children pretty soon and is looking your way for the ideal family man.”
Now you have a good idea of what your user wants you can match them with the twin that suits them best. So on this occasion, don’t send Tracy on a date with the rogue twin who likes to travel the world. Tracy isn’t looking for that. Instead, send the homely twin who wants to settle down soon with a family. See where we are coming from? This is really, really useful information. With this you can analyse what attracted Tracy to your twins in the first place and how to repeat/avoid this connection in the future.
VoC Testing takes this one step further. Similar to the above, this process intercepts users and collects feedback, but after they have experienced your site or product. So this is like asking your user how their date went and asking them to provide feedback on the attractiveness of Twin 1 or 2 (depending on which variant they were provided with). Who doesn’t want to find out how a date has gone? Everyone would appreciate a little date feedback, especially when it’s frank and honest!
With a VoC test you will find out what your user felt about the date, the problems they encountered and the likeliness of the user attending a second date. Now everything is completely transparent we know:
As the evidence clearly shows, A/B Testing alone is a handy tool but is not as comprehensive as it can be when you introduce other multivariate testing methods on top. By simply testing A and B against each other you can find out which variant works best – but you won’t exactly know why. By introducing other multivariate methods, such as True Intent Studies and VoC, you are able to find out so much more.
The guys over at User Zoom know there stuff when it comes to this kind of thing! Check out their website for more information on multi-variate testing.
