Related papers: Controlled Experimentation in Continuous Experimen…
Over the past two decades the field of computational science and engineering (CSE) has penetrated both basic and applied research in academia, industry, and laboratories to advance discovery, optimize systems, support decision-makers, and…
With the growing needs of online A/B testing to support the innovation in industry, the opportunity cost of running an experiment becomes non-negligible. Therefore, there is an increasing demand for an efficient continuous monitoring…
Innovations across science and industry are evaluated using randomized trials (a.k.a. A/B tests). While simple and robust, such static designs are inefficient or infeasible for testing many hypotheses. Adaptive designs can greatly improve…
Drawing on ideas from continuous integration, we present concepts of an automated benchmarking pipeline for high performance applications. Customization and collaboration have been key design goals owing to the requirements of…
A vast array of transformative technologies developed over the past decade has enabled measurement and perturbation at ever increasing scale, yet our understanding of many systems remains limited by experimental capacity. Overcoming this…
Science and technology have a growing need for effective mechanisms that ensure reliable, controlled performance from black-box machine learning algorithms. These performance guarantees should ideally hold conditionally on the input-that is…
Scientific fact-checking aims to determine the veracity of scientific claims by retrieving and analysing evidence from research literature. The problem is inherently more complex than general fact-checking since it must accommodate the…
Context: Context-aware contemporary software systems (CACSS) are mainstream. Furthermore, they present challenges for current engineering practices. These challenges are distinctively present when testing CACSS, as the variation of context…
Computer Science (CS) education is expanding rapidly, but educators continue to face persistent challenges in teaching and learning environments.Despite growing interest, limited systematic work exists to categorize and synthesize the…
A common concern in experimental research is the auditability and reproducibility of experiments. Experiments are usually designed, provisioned, managed, and analyzed by diverse teams of specialists (e.g., researchers, technicians and…
Compilers are widely-used infrastructures in accelerating the software development, and expected to be trustworthy. In the literature, various testing technologies have been proposed to guarantee the quality of compilers. However, there…
The measurement of human behavior remains a central challenge across the behavioral sciences. Traditional approaches typically rely on passive observation of responses collected under static or weakly controlled conditions, limiting the…
Experimental science is enabled by the combination of synthesis, imaging, and functional characterization. Synthesis of a new material is typically followed by a set of characterization methods aiming to provide feedback for optimization or…
Context: Modern Systems of Systems (SoSs) increasingly operate in dynamic environments (e.g., smart cities, autonomous vehicles) where runtime composition -- the on-the-fly discovery, integration, and coordination of constituent systems…
The operation of instruments and detectors in laboratory or beamline environments presents a complex challenge, requiring stable operation of multiple concurrent devices, often controlled by separate hardware and software solutions. These…
Learning from demonstration is one of the most promising methods to counteract the challenging long-term trends in repetitive industrial assembly. It offers not only a programming technique that is accessible to workers on the shop floor,…
The potential benefits of autonomous systems have been driving intensive development of such systems, and of supporting tools and methodologies. However, there are still major issues to be dealt with before such development becomes…
With the extensive use of digital devices, online experimental platforms are commonly used to conduct experiments to collect data for evaluating different variations of products, algorithms, and interface designs, a.k.a., A/B tests. In…
Background. Reproducibility is essential to the scientific method, but reproduction is often a laborious task. Recent works have attempted to automate this process and relieve researchers of this workload. However, due to varying…
We study the constrained linear quadratic regulator with unknown dynamics, addressing the tension between safety and exploration in data-driven control techniques. We present a framework which allows for system identification through…