Related papers: Optimal Testing in the Experiment-rich Regime
We address the problem of A/B testing, a widely used protocol for evaluating the potential improvement achieved by a new decision system compared to a baseline. This protocol segments the population into two subgroups, each exposed to a…
In an empirical comparisons of algorithms we might compare run times over a set of benchmark problems to decide which one is fastest, i.e. an algorithmic horse race. Ideally we would like to download source code for the algorithms, compile…
The last decade witnessed an explosion in the availability of data for operations research applications. Motivated by this growing availability, we propose a novel schema for utilizing data to design uncertainty sets for robust optimization…
This paper traces the strong relations between experimental design and control, such as the use of optimal inputs to obtain precise parameter estimation in dynamical systems and the introduction of suitably designed perturbations in…
Traditional hypothesis tests for differences between binomial proportions are at risk of being too liberal (Wald test) or overly conservative (Fisher's exact test). This problem is exacerbated in small samples. Regulators favour exact…
A problem of a new physical model test given observed experimental data is a typical one for modern experiments of high energy physics (HEP). A solution of the problem may be provided with two alternative statistical formalisms, namely…
The modeling of complex systems such as ecological or socio-economic systems can be very challenging. Although various modeling approaches exist, they are generally not compatible and mutually consistent, and empirical data often do not…
The note studies the problem of selecting a good enough subset out of a finite number of alternatives under a fixed simulation budget. Our work aims to maximize the posterior probability of correctly selecting a good subset. We formulate…
The existence of a (p-)optimal propositional proof system is a major open question in (proof) complexity; many people conjecture that such systems do not exist. Krajicek and Pudlak (1989) show that this question is equivalent to the…
A/B testing, or online experiment is a standard business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries. Major challenges arise in online experiments of two-sided marketplace…
Randomized A/B comparisons of alternative pedagogical strategies or other course improvements could provide useful empirical evidence for instructor decision-making. However, traditional experiments do not provide a straightforward pathway…
Empirical research on meta-algorithmics, such as algorithm selection, configuration, and scheduling, often relies on extensive and thus computationally expensive experiments. With the large degree of freedom we have over our experimental…
Randomized experiments have been the gold standard for assessing the effectiveness of a treatment or policy. The classical complete randomization approach assigns treatments based on a prespecified probability and may lead to inefficient…
With the current trend in Model-Based Systems Engineering towards Digital Engineering and early Validation & Verification, experiments are increasingly used to estimate system parameters and explore design decisions. Managing such…
In the classical experimental design setting, an experimenter E has access to a population of $n$ potential experiment subjects $i\in \{1,...,n\}$, each associated with a vector of features $x_i\in R^d$. Conducting an experiment with…
While most heuristics studied in heuristic search depend only on the state, some accumulate information during search and thus also depend on the search history. Various existing approaches use such dynamic heuristics in $\mathrm{A}^*$-like…
Reinforcement learning can greatly benefit from the use of options as a way of encoding recurring behaviours and to foster exploration. An important open problem is how can an agent autonomously learn useful options when solving particular…
Several systematic studies have suggested that a large fraction of published research is not reproducible. One probable reason for low reproducibility is insufficient sample size, resulting in low power and low positive predictive value. It…
Combining test statistics from independent trials or experiments is a popular method of meta-analysis. However, there is very limited theoretical understanding of the power of the combined test, especially in high-dimensional models…
Bayesian experimental design involves the optimal allocation of resources in an experiment, with the aim of optimising cost and performance. For implicit models, where the likelihood is intractable but sampling from the model is possible,…