Related papers: AExGym: Benchmarks and Environments for Adaptive E…

We develop a theoretical framework for sample splitting in A/B testing environments, where data for each test are partitioned into two splits to measure methodological performance when the true impacts of tests are unobserved. We show that…

Traditional statistical analysis requires that the analysis process and data are independent. By contrast, the new field of adaptive data analysis hopes to understand and provide algorithms and accuracy guarantees for research as it is…

Nonlinear, adaptive, or otherwise complex control techniques are increasingly relied upon to ensure the safety of systems operating in uncertain environments. However, the nonlinearity of the resulting closed-loop system complicates…

The construction of decision-theoretic Bayesian designs for realistically-complex nonlinear models is computationally challenging, as it requires the optimization of analytically intractable expected utility functions over high-dimensional…

Optimal experiment design for parameter estimation is a research topic that has been in the interest of various studies. A key problem in optimal input design is that the optimal input depends on some unknown system parameters that are to…

The challenge of optimal design of experiments (DOE) pervades materials science, physics, chemistry, and biology. Bayesian optimization has been used to address this challenge in vast sample spaces, although it requires framing experimental…

Evaluating Software testability can assist software managers in optimizing testing budgets and identifying opportunities for refactoring. In this paper, we abandon the traditional approach of pursuing testability measurements based on the…

In the experimental design literature, Neyman allocation refers to the practice of allocating units into treated and control groups, potentially in unequal numbers proportional to their respective standard deviations, with the objective of…

We develop asymptotic approximations that can be applied to sequential estimation and inference problems, adaptive randomized controlled trials, and related settings. In batched adaptive settings where the decision at one stage can affect…

Benchmarking involves designing, running and disseminating rigorous performance assessments of methods, most often for data analysis and software tools, but the process can also be applied to experimental systems. Ideally, a benchmarking…

Modal testing plays a critical role in structural analysis by providing essential insights into dynamic behaviour across a wide range of engineering industries. In practice, designing an effective modal test campaign involves complex…

Reasoning has emerged as the next major frontier for language models (LMs), with rapid advances from both academic and industrial labs. However, this progress often outpaces methodological rigor, with many evaluations relying on…

A/B testing has become a gold standard for modern technological companies to conduct policy evaluation. Yet, its application to time series experiments, where policies are sequentially assigned over time, remains challenging. Existing…

In recent years, the need for neutral benchmark studies that focus on the comparison of methods from computational sciences has been increasingly recognised by the scientific community. While general advice on the design and analysis of…

Adaptive systems react to changes in their environment by changing their behavior. Identifying these needed adaptations is very difficult, but central to requirements elicitation for adaptive systems. As the necessary or potential…

We introduce TimeSeriesGym, a scalable benchmarking framework for evaluating Artificial Intelligence (AI) agents on time series machine learning engineering challenges. Existing benchmarks lack scalability, focus narrowly on model building…

The rigorous evaluation of the novelty of a scientific paper is, even for human scientists, a challenging task. With the increasing interest in AI scientists and AI involvement in scientific idea generation and paper writing, it also…

Simulators are a critical component of modern robotics research. Strategies for both perception and decision making can be studied in simulation first before deployed to real world systems, saving on time and costs. Despite significant…

In nonstandard testing environments, researchers often derive ad hoc tests with correct (asymptotic) size, but their optimality properties are typically unknown a priori and difficult to assess. This paper develops a numerical framework for…

Adaptability is a significant property which enables software systems to continuously provide the required functionality and achieve optimal performance. The recognised importance of adaptability makes its evaluation an essential task.…