Related papers: Number of Repetitions in Re-randomization Tests
Considerable interest has recently been focused on studying multiple phenotypes simultaneously in both epidemiological and genomic studies, either to capture the multidimensionality of complex disorders or to understand shared etiology of…
We introduce the notion of a reproducible algorithm in the context of learning. A reproducible learning algorithm is resilient to variations in its samples -- with high probability, it returns the exact same output when run on two samples…
Sequential Multiple-Assignment Randomized Trials (SMARTs) play an increasingly important role in psychological and behavioral health research. This experimental approach enables researchers to answer scientific questions about how to…
With the growth in experimental studies in education, policymakers and practitioners are interested in understanding not only what works, but for whom an intervention works. This interest in the generalizability of a study's findings has…
Accurate uncertainty estimates can significantly improve the performance of iterative design of experiments, as in Sequential and Reinforcement learning. For many such problems in engineering and the physical sciences, the design task…
There is an increasing number of potential biomarkers that could allow for early assessment of treatment response or disease progression. However, measurements of quantitative biomarkers are subject to random variability. Hence, differences…
Random testing approaches work by generating inputs at random, or by selecting inputs randomly from some pre-defined operational profile. One long-standing question that arises in this and other testing contexts is as follows: When can we…
Participants in clinical trials are often viewed as a unique, finite population. Yet, statistical analyses often assume that participants were randomly sampled from a larger population. Under Complete Randomization, Randomization-Based…
In randomized experiments, treatment and control groups should be roughly the same--balanced--in their distributions of pretreatment variables. But how nearly so? Can descriptive comparisons meaningfully be paired with significance tests?…
Consistently checking the statistical significance of experimental results is one of the mandatory methodological steps to address the so-called "reproducibility crisis" in deep reinforcement learning. In this tutorial paper, we explain how…
The win ratio is increasingly used in randomized trials due to its intuitive clinical interpretation, ability to incorporate the relative importance of composite endpoints, and its capacity for combining different types of outcomes (e.g.…
Restricting randomization in the design of experiments (e.g., using blocking/stratification, pair-wise matching, or rerandomization) can improve the treatment-control balance on important covariates and therefore improve the estimation of…
In solving simulation-based stochastic root-finding or optimization problems that involve rare events, such as in extreme quantile estimation, running crude Monte Carlo can be prohibitively inefficient. To address this issue, importance…
The paper presents an investigation of estimating treatment effect using different matching methods. The study proposed a new method which is computationally efficient and convenient in implication-'largest caliper matching' and compared…
LLM leaderboards often rely on single stochastic runs, but how many repetitions are required for reliable conclusions remains unclear. We re-evaluate eight state-of-the-art models on the AI4Math Benchmark with three independent runs per…
The use and development of mobile interventions are experiencing rapid growth. In "just-in-time" mobile interventions, treatments are provided via a mobile device and they are intended to help an individual make healthy decisions "in the…
The correlation between a random sequence and its transformed sequences is studied. In the case of a permutation operation or, in other word, the shuffling operation, it is shown that the correlation can be so small that the sequences can…
A new method based on the rejection sampling for finding statistical tests is proposed. This method is conceptually intuitive, easy to implement, and applicable for arbitrary dimension. To illustrate its potential applicability, three…
A multivariate quantile regression model with a factor structure is proposed to study data with many responses of interest. The factor structure is allowed to vary with the quantile levels, which makes our framework more flexible than the…
Revisions are almost entirely in the introduction and conclusion. Results are unchanged, however the comments and recommendations on different generators were changed, and more references were added.