Related papers: Setting the duration of online A/B experiments
Indirect experiments provide a valuable framework for estimating treatment effects in situations where conducting randomized control trials (RCTs) is impractical or unethical. Unlike RCTs, indirect experiments estimate treatment effects by…
We consider the problem of forming prediction sets in an online setting where the distribution generating the data is allowed to vary over time. Previous approaches to this problem suffer from over-weighting historical data and thus may…
Tests for paired censored outcomes have been extensively studied, with some justified in the context of randomization-based inference. These tests are primarily designed to detect an overall treatment effect across the entire follow-up…
We study ratio metrics in A/B testing at the presence of correlation among observations coming from the same user and provides practical guidance especially when two metrics contradict each other. We propose new estimating methods to…
Popular statistical software provides Bayesian information criterion (BIC) for multilevel models or linear mixed models. However, it has been observed that the combination of statistical literature and software documentation has led to…
Meta-analyses are commonly used to provide solid evidence across numerous studies. Traditional moment methods, such as the DerSimonian-Laird method, remain popular in spite of the availability of more accurate alternatives. While moment…
Online experimentation platforms collect user feedback at low cost and large scale. Some systems even support real-time or near real-time data processing, and can update metrics and statistics continuously. Many commonly used metrics, such…
This paper studies how to design two-wave experiments in the presence of spillovers for precise inference on treatment effects. We consider units connected through a single network, local dependence among individuals, and a general class of…
In online platforms, the impact of a treatment on an observed outcome may change over time as 1) users learn about the intervention, and 2) the system personalization, such as individualized recommendations, change over time. We introduce a…
Population-adjusted indirect comparisons estimate treatment effects when access to individual patient data is limited and there are cross-trial differences in effect modifiers. Popular methods include matching-adjusted indirect comparison…
Background: When developing a clinical prediction model using time-to-event data, previous research focuses on the sample size to minimise overfitting and precisely estimate the overall risk. However, instability of individual-level risk…
We compute bias, variance, and approximate confidence intervals for the efficiency of a random selection process under various special conditions that occur in practical data analysis. We consider the following cases: a) the number of…
Understanding how the causal effect of a treatment evolves over time, including the potential for waning, is important for informed decisions on treatment discontinuation or repetition. For example, waning vaccine protection influences…
Adapting the final sample size of a trial to the evidence accruing during the trial is a natural way to address planning uncertainty. Designs with adaptive sample size need to account for their optional stopping to guarantee strict type-I…
A/B testing plays a central role in data-driven product development, guiding launch decisions for new features and designs. However, treatment effect estimates are often noisy due to short horizons, early stopping, and slowly accumulating…
Stepped wedge cluster randomized controlled trials are typically analyzed using models that assume the full effect of the treatment is achieved instantaneously. We provide an analytical framework for scenarios in which the treatment effect…
In this paper we examine rigorously the evidence for dependence among data size, transfer rate and duration in Internet flows. We emphasize two statistical approaches for studying dependence, including Pearson's correlation coefficient and…
By combining a bound on the absolute value of the difference of mutual information between two joint probablity distributions with a fixed variational distance, and a bound on the probability of a maximal deviation in variational distance…
The aim of this paper is to establish a causal link between the policies implemented by technology companies and the outcomes they yield within intricate temporal and/or spatial dependent experiments. We propose a novel…
The problem of evaluating the effectiveness of a treatment or policy commonly appears in causal inference applications under network interference. In this paper, we suggest the new method of high-dimensional network causal inference (HNCI)…