Related papers: Assign Experiment Variants at Scale in Online Cont…
A/B testing plays a central role in data-driven product development, guiding launch decisions for new features and designs. However, treatment effect estimates are often noisy due to short horizons, early stopping, and slowly accumulating…
e consider the experimental design problem in an online environment, an important practical task for reducing the variance of estimates in randomized experiments which allows for greater precision, and in turn, improved decision making. In…
Rerandomization is a modern experimental design technique that repeatedly randomizes treatment assignments until covariates are deemed balanced between treatment groups. This enhances the precision and coherence of causal effect estimators,…
Online controlled experiments (A/B tests) are fundamental to data-driven decision-making in the digital economy. However, their real-world application is frequently compromised by two critical shortcomings: the use of statistically flawed…
Randomized trials, also known as A/B tests, are used to select between two policies: a control and a treatment. Given a corresponding set of features, we can ideally learn an optimized policy P that maps the A/B test data features to action…
Usually, managers or technical leaders in software projects assign issues manually. This task may become more complex as more detailed is the issue description. This complexity can also make the process more prone to errors (misassignments)…
A/B testing has become the gold standard for policy evaluation in modern technological industries. Motivated by the widespread use of switchback experiments in A/B testing, this paper conducts a comprehensive comparative analysis of various…
Sampling algorithms play a pivotal role in probabilistic AI. However, verifying if a sampler program indeed samples from the claimed distribution is a notoriously hard problem. Provably correct testers like Barbarik, Teq, Flash, CubeProbe…
AB testing aids business operators with their decision making, and is considered the gold standard method for learning from data to improve digital user experiences. However, there is usually a gap between the requirements of practitioners,…
Estimation of social influence in networks can be substantially biased in observational studies due to homophily and network correlation in exposure to exogenous events. Randomized experiments, in which the researcher intervenes in the…
As any scientific discipline, the software engineering (SE) research community strives to contribute to the betterment of the target population of our research: software producers and consumers. We will only achieve this betterment if we…
We explain and provide examples of a formalism that supports the methodology of discovering how to adapt and personalize technology by combining randomized experiments with variables associated with user models. We characterize a formal…
In this paper, we study randomized methods for feedback design of uncertain systems. The first contribution is to derive the sample complexity of various constrained control problems. In particular, we show the key role played by the…
This paper proposes an adaptive random experiment design (ARED) algorithm that can be applied to optimize the multiple factors and levels experiments. The algorithm takes real-time model error as the adaptive condition, and outputs a model…
Randomized controlled experiments assess new policy impacts on performance metrics to inform launch decisions. Traditional approaches evaluate metrics independently despite correlations, and mixed results (e.g., positive revenue impact,…
Randomized benchmarking is an experimental procedure intended to demonstrate control of quantum systems. The procedure extracts the average error introduced by a set of control operations. When the target set of operations is intended to be…
Randomized experiments (A/B testings) have become the standard way for web-facing companies to guide innovation, evaluate new products, and prioritize ideas. There are times, however, when running an experiment is too complicated (e.g., we…
Randomized benchmarking is a widely used experimental technique to characterize the average error of quantum operations. Benchmarking procedures that scale to enable characterization of $n$-qubit circuits rely on efficient procedures for…
The treatment assignment mechanism in a randomized clinical trial can be optimized for statistical efficiency within a specified class of randomization mechanisms. Optimal designs of this type have been characterized in terms of the…
We have seen a massive growth of online experiments at LinkedIn, and in industry at large. It is now more important than ever to create an intelligent A/B platform that can truly democratize A/B testing by allowing everyone to make quality…