Related papers: Informed Bayesian Inference for the A/B Test
It is often claimed that Bayesian methods, in particular Bayes factor methods for hypothesis testing, can deal with optional stopping. We first give an overview, using elementary probability theory, of three different mathematical meanings…
Bayesian statistics is an integral part of contemporary applied science. bayesics provides a single framework, unified in syntax and output, for performing the most commonly used statistical procedures, ranging from one- and two-sample…
We present a new test of hypothesis in which we seek the probability of the null conditioned on the data, where the null is a simplification undertaken to counter the intractability of the more complex model, that the simpler null model is…
This article proposes different tests for treatment effect heterogeneity when the outcome of interest, typically a duration variable, may be right-censored. The proposed tests study whether a policy 1) has zero distributional (average)…
Observational studies are often conducted to estimate causal effects of treatments or exposures on event-time outcomes. Since treatments are not randomized in observational studies, techniques from causal inference are required to adjust…
Online controlled experiments, colloquially known as A/B-tests, are the bread and butter of real-world recommender system evaluation. Typically, end-users are randomly assigned some system variant, and a plethora of metrics are then…
A/B tests, also known as randomized controlled experiments (RCTs), are the gold standard for evaluating the impact of new policies, products, or decisions. However, these tests can be costly in terms of time and resources, potentially…
The average treatment effect, which is the difference in expectation of the counterfactuals, is probably the most popular target effect in causal inference with binary treatments. However, treatments may have effects beyond the mean, for…
Randomized experiments (A/B testings) have become the standard way for web-facing companies to guide innovation, evaluate new products, and prioritize ideas. There are times, however, when running an experiment is too complicated (e.g., we…
A sequence of null hypotheses regarding the number of negligible effects (zero effects) in orthogonal saturated designs is formulated. Two step-up simultaneous testing procedures are proposed to identify active effects (nonzero effects)…
In several large-scale replication projects, statistically non-significant results in both the original and the replication study have been interpreted as a "replication success". Here we discuss the logical problems with this approach:…
The Bayes factor, the data-based updating factor of the prior to posterior odds of two hypotheses, is a natural measure of statistical evidence for one hypothesis over the other. We show how Bayes factors can also be used for parameter…
In this paper, a Bayesian approach is developed for simultaneously comparing multiple experimental treatments with a common control treatment in an exploratory clinical trial. The sample size is set to ensure that, at the end of the study,…
A/B test, a simple type of controlled experiment, refers to the statistical procedure of experimenting to compare two treatments applied to test subjects. For example, many IT companies frequently conduct A/B tests on their users who are…
Studies using assays to quantify the expression of thousands of genes on tens to thousands of cell samples have been carried out for over 20 years. Such assays are based on microarrays, DNA sequencing or other molecular technologies. All…
In observational studies, instrumental variables estimation is greatly utilized to identify causal effects. One of the key conditions for the instrumental variables estimator to be consistent is the exclusion restriction, which indicates…
There has been strong recent interest in testing interval null hypothesis for improved scientific inference. For example, Lakens et al (2018) and Lakens and Harms (2017) use this approach to study if there is a pre-specified meaningful…
This paper introduces an overidentification test of two alternative assumptions to identify the average treatment effect on the treated in a two-period panel data setting: unconfoundedness and common trends. Under the unconfoundedness…
Disease models are used to examine the likely impact of therapies, interventions and public policy changes. Ensuring that these are well calibrated on the basis of available data and that the uncertainty in their projections is properly…
Testing the equality of two proportions is a common procedure in science, especially in medicine and public health. In these domains it is crucial to be able to quantify evidence for the absence of a treatment effect. Bayesian hypothesis…