Then, we use a statistical method to determine which variant is better. In any A/B test, we use the data we collect from variants A and B to compute some metric for each variant (e.g. In this post, I’ll cover the basics of experimentation, present the benefits of Bayesian A/B testing, and discuss the nuances of using it effectively. Bayesian A/B testing accomplishes this without sacrificing reliability by controlling the magnitude of our bad decisions instead of the false positive rate. By using Bayesian A/B testing over the course of many experiments, we can accumulate the gains from many incremental improvements. In experiments where the improvement of the new variant is small, Bayesian methodology is more willing to accept the new variant. Given our use case of continuous iteration, we find that Bayesian A/B testing better balances risk and speed. We switched to an A/B testing framework that uses Bayesian statistics because it allows us to innovate faster and improve more. The data science team at Convoy believes that the frequentist methodology of experimentation isn’t ideal for product innovation.

Bayes test models