Conducting A/B Experiments with a Scalable Architecture
Abstract
A/B experiments are commonly used in research to compare the effects of changing one or more variables in two different experimental groups - a control group and a treatment group. While the benefits of using A/B experiments are widely known and accepted, there is less agreement on a principled approach to creating software infrastructure systems to assist in rapidly conducting such experiments. We propose a four-principle approach for developing a software architecture to support A/B experiments that is domain agnostic and can help alleviate some of the resource constraints currently needed to successfully implement these experiments: the software architecture (i) must retain the typical properties of A/B experiments, (ii) capture problem solving activities and outcomes, (iii) allow researchers to understand the behavior and outcomes of participants in the experiment, and (iv) must enable automated analysis. We successfully developed a software system to encapsulate these principles and implement it in a real-world A/B experiment.
Cite
@article{arxiv.2309.13450,
title = {Conducting A/B Experiments with a Scalable Architecture},
author = {Andrew Hornback and Sungeun An and Scott Bunin and Stephen Buckley and John Kos and Ashok Goel},
journal= {arXiv preprint arXiv:2309.13450},
year = {2023}
}