Related papers: YEAST: Yet Another Sequential Test
We propose a new approach to sequential testing which is an adaptive (on-line) extension of the (off-line) framework developed in [10]. It relies upon testing of pairs of hypotheses in the case where each hypothesis states that the vector…
Modern application of A/B tests is challenging due to its large scale in various dimensions, which demands flexibility to deal with multiple testing sequentially. The state-of-the-art practice first reduces the observed data stream to…
Randomized A/B tests within online learning platforms represent an exciting direction in learning sciences. With minimal assumptions, they allow causal effect estimation without confounding bias and exact statistical inference even in small…
In most prediction and estimation situations, scientists consider various statistical models for the same problem, and naturally want to select amongst the best. Hansen et al. (2011) provide a powerful solution to this problem by the…
Accelerated life testing (ALT) is typically used to assess the reliability of material's lifetime under desired stress levels. Recent advances in material engineering have made a variety of material alternatives readily available. To…
We propose a nonparametric sequential test that aims to address two practical problems pertinent to online randomized experiments: (i) how to do a hypothesis test for complex metrics; (ii) how to prevent type $1$ error inflation under…
While there exists a large amount of literature on the general challenges of and best practices for trustworthy online A/B testing, there are limited studies on sample size estimation, which plays a crucial role in trustworthy and efficient…
We propose a new algorithmic framework for sequential hypothesis testing with i.i.d. data, which includes A/B testing, nonparametric two-sample testing, and independence testing as special cases. It is novel in several ways: (a) it takes…
We introduce sequential analysis in quantum information processing, by focusing on the fundamental task of quantum hypothesis testing. In particular our goal is to discriminate between two arbitrary quantum states with a prescribed error…
We develop a theoretical framework for sample splitting in A/B testing environments, where data for each test are partitioned into two splits to measure methodological performance when the true impacts of tests are unobserved. We show that…
Large-scale online platforms and marketplace systems often evaluate new policies through experiments that randomize treatment across operational units (e.g., geographies, regions, or clusters) over many time periods. In these settings,…
In this paper, we propose a general method for testing composite hypotheses. Our idea is to use confidence limits to define stopping and decision rules. The requirements of operating characteristic function can be satisfied by adjusting the…
Financial statement auditing is conducted under a risk-based evidence approach to obtain reasonable assurance. In practice, auditors often perform additional sampling or related procedures when an initial sample does not provide a…
Participants in online experiments often enroll over time, which can compromise sample representativeness due to temporal shifts in covariates. This issue is particularly critical in A/B tests, online controlled experiments extensively used…
Bayesian A/B testing investigates metric changes using the joint posterior distribution of two (or more) experimentally-derived datasets. The construction of said joint posterior is often a time-consuming process requiring specialized…
We consider the problem of sequentially testing for changes in the mean parameter of a time series, compared to a benchmark period. Most tests in the literature focus on the null hypothesis of a constant mean versus the alternative of a…
We present a sequential testing method to identify a practically significant effect. We build on the existing mixture sequential probability ratio test (mSPRT) that can sequentially test for a non-zero treatment effect by using a truncated…
AB testing evaluates the difference between a control and a treatment in a statistically rigorous manner. Continuous monitoring allows statistical evaluation of an AB test as it proceeds. One goal of continuous monitoring is early stopping…
Sequential monitoring in clinical trials is often employed to allow for early stopping and other interim decisions, while maintaining the type I error rate. However, sequential monitoring is typically described only in the context of a…
Online experiments such as Randomised Controlled Trials (RCTs) or A/B-tests are the bread and butter of modern platforms on the web. They are conducted continuously to allow platforms to estimate the causal effect of replacing system…