Related papers: Randomness as Reference: Benchmark Metric for Opti…
In large-scale benchmarking of stochastic optimization algorithms, the key challenge is no longer whether repeated runs are needed for reliability, but how to determine when sufficient evidence has been collected without incurring…
The growing need for synthetic time series, due to data augmentation or privacy regulations, has led to numerous generative models, frameworks, and evaluation measures alike. Objectively comparing these measures on a large scale remains an…
Software architecture is the foundation of a system's ability to achieve various quality attributes, including software performance. However, there lacks comprehensive and in-depth understanding of why and how software architecture and…
The experimental evaluation of algorithms results in a large set of data which generally do not follow a normal distribution or are not heteroscedastic. Besides, some of its entries may be missing, due to the inability of an algorithm to…
In simulation-based optimization, the optimal setting of the input parameters of the objective function can be determined by heuristic optimization techniques. However, when simulators model the stochasticity of real-world problems, their…
How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing? We introduce ALE-Bench, a new…
Bayesian optimization has emerged as a highly effective tool for the safe online optimization of systems, due to its high sample efficiency and noise robustness. To further enhance its efficiency, reduced physical models of the system can…
Covariate adjustment is widely recommended to improve statistical efficiency in randomized clinical trials (RCTs), yet empirical evidence comparing available strategies remains limited. This lack of real-world evaluation leaves unresolved…
Sampling-based planning algorithms are the most common probabilistically complete algorithms and are widely used on many robot platforms. Within this class of algorithms, many variants have been proposed over the last 20 years, yet there is…
Executing various sequences of system functions in a system under test represents one of the primary techniques in software testing. The natural way to create effective, consistent and efficient test sequences is to model the system under…
Metaheuristic algorithms are methods devised to efficiently solve computationally challenging optimization problems. Researchers have taken inspiration from various natural and physical processes alike to formulate meta-heuristics that have…
This material provides thorough tutorials on some optimization techniques frequently used in various engineering disciplines, including convex optimization, linearization techniques and mixed-integer linear programming, robust optimization,…
Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-based systems, but developers nonetheless claim that their…
Characterization of experimental systems is an essential step in developing and improving quantum hardware. A collection of protocols known as Randomized Benchmarking (RB) was developed in the past decade, which provides an efficient way to…
During the past sixty years, a lot of effort has been made regarding the productive efficiency. Such endeavours provided an extensive bibliography on this subject, culminating in two main methods, named the Stochastic Frontier Analysis…
Graph pooling has gained attention for its ability to obtain effective node and graph representations for various downstream tasks. Despite the recent surge in graph pooling approaches, there is a lack of standardized experimental settings…
AI workloads, particularly those driven by deep learning, are introducing novel usage patterns to high-performance computing (HPC) systems that are not comprehensively captured by standard HPC benchmarks. As one of the largest academic…
Ignoring uncertainty in combinatorial optimization leads to suboptimal decisions in practice. Nevertheless, the focus is often on deterministic combinatorial optimization problems, mainly because they are already challenging enough without…
Several different ways exist for approaching hard optimization problems. Mathematical programming techniques, including (integer) linear programming-based methods and metaheuristic approaches, are two highly successful streams for…
Benchmarks are crucial to measuring and steering progress in artificial intelligence (AI). However, recent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and…