A/B Testing or Multi-Armed Bandit?

There’s been a lot of talk on the web recently about the Multi-Armed Bandit method of testing new web designs being superior to standard A/B testing. For those who aren’t familiar, Multi-Armed Bandit method tests with only 10% of your traffic, and for the other 90% uses whatever the currently best performing variation is. The actual percentages are adjustable of course, but the idea is you test with less of your traffic and convert on more of it.

Of course, this is a great idea but the problem is the majority of sites on the web don’t get enough traffic for 10% of it to result in a statistically significant number quickly. If you’re testing over a long period of time, things can vary greatly, and you no longer have as solid of a control to compare against (since your control is possibly only getting a portion of that 10% as well.) The 90% isn’t usable in the statistical comparison since which variation is getting 90% of the traffic can change over the course of the test.

The bottom line is standard A/B testing is probably better suited for smaller web sites. It’s also best for those interested in concluding the test quickly and for having solid conclusions to base a plan of action on. However, if you have a high traffic website and want to be constantly testing multiple variations, and using whatever variations perform best, then Multi-Armed Bandit can do just that. Multi-Armed is also great for reducing risk when testing radical design changes because you’re only showing it to a small percentage of your traffic.

As usual, tools are tools and it’s up to the informed and thoughtful developer to use them appropriately to create the best results for herself or her clients.