Bounds on Bayes Factors for Binomial A/B Testing
Other Statistics
2019-03-04 v1 Machine Learning
Abstract
Bayes factors, in many cases, have been proven to bridge the classic -value based significance testing and bayesian analysis of posterior odds. This paper discusses this phenomena within the binomial A/B testing setup (applicable for example to conversion testing). It is shown that the bayes factor is controlled by the \emph{Jensen-Shannon divergence} of success ratios in two tested groups, which can be further bounded by the Welch statistic. As a result, bayesian sample bounds almost match frequentionist's sample bounds. The link between Jensen-Shannon divergence and Welch's test as well as the derivation are an elegant application of tools from information geometry.
Keywords
Cite
@article{arxiv.1903.00049,
title = {Bounds on Bayes Factors for Binomial A/B Testing},
author = {Maciej Skorski},
journal= {arXiv preprint arXiv:1903.00049},
year = {2019}
}