Related papers: Seesaw Experimentation: A/B Tests with Spillovers

In randomised trials, continuous endpoints are often measured with some degree of error. This study explores the impact of ignoring measurement error, and proposes methods to improve statistical inference in the presence of measurement…

This study considers testing the specification of spillover effects in causal inference. We focus on experimental settings in which the treatment assignment mechanism is known to researchers. We develop a new randomization test utilizing a…

We study the design and analysis of switchback experiments conducted on a single aggregate unit. The design problem is to partition the continuous time space into intervals and switch treatments between intervals, in order to minimize the…

An interesting empirical result in the assembly line literature states that slightly unbalanced assembly lines (in the format of a bowl - with central stations less loaded than the external ones) present higher throughputs than perfectly…

In this paper, we draw attention to a problem that is often overlooked or ignored by companies practicing hypothesis testing (A/B testing) in online environments. We show that conducting experiments on limited inventory that is shared…

The problem of simultaneously testing the marginal distributions of sequentially monitored, independent data streams is considered. The decisions for the various testing problems can be made at different times, using data from all streams,…

A/B tests are often required to be conducted on subjects that might have social connections. For e.g., experiments on social media, or medical and social interventions to control the spread of an epidemic. In such settings, the SUTVA…

Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of…

Online controlled experiments, colloquially known as A/B-tests, are the bread and butter of real-world recommender system evaluation. Typically, end-users are randomly assigned some system variant, and a plethora of metrics are then…

Much research on Machine Learning testing relies on empirical studies that evaluate and show their potential. However, in this context empirical results are sensitive to a number of parameters that can adversely impact the results of the…

Readout of superconducting qubits faces a trade-off between measurement speed and unwanted back-action on the qubit caused by the readout drive, such as $T_1$ degradation and leakage out of the computational subspace. The readout is…

In estimating spillover effects under network interference, practitioners often use linear regression with either the number or fraction of treated neighbors as regressors. An often overlooked fact is that the latter is undefined for units…

Online experiments such as Randomised Controlled Trials (RCTs) or A/B-tests are the bread and butter of modern platforms on the web. They are conducted continuously to allow platforms to estimate the causal effect of replacing system…

Errors-in-variables is a long-standing, difficult issue in linear regression; and progress depends in part on new identifying assumptions. I characterize measurement error as bad-leverage points and assume that fewer than half the sample…

Group testing is utilized in the case when we want to find a few defectives among large amount of items. Testing n items one by one requires n tests, but if the ratio of defectives is small, group testing is an efficient way to reduce the…

Estimating software testability can crucially assist software managers to optimize test budgets and software quality. In this paper, we propose a new approach that radically differs from the traditional approach of pursuing testability…

Standard A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously, while adaptive experimental designs (AED) alone fall short in inferring experiment statistics such as the average…

Randomized A/B tests within online learning platforms represent an exciting direction in learning sciences. With minimal assumptions, they allow causal effect estimation without confounding bias and exact statistical inference even in small…

Network interference amounts to the treatment status of one unit affecting the potential outcome of other units in the population. Testing for spillover effects in this setting makes the null hypothesis non-sharp. An interesting approach to…

Technology firms conduct randomized controlled experiments ("A/B tests") to learn which actions to take to improve business outcomes. In firms with mature experimentation platforms, experimentation programs can consist of many thousands of…