Related papers: Logical Randomized Benchmarking
Among the most popular and well studied quantum characterization, verification and validation techniques is randomized benchmarking (RB), an important statistical tool used to characterize the performance of physical logic operations useful…
Numerous mitigation methods exist for quantum noise suppression, making it challenging to identify the optimum approach for a specific application; especially as ongoing advances in hardware tuning and error correction are expected to…
Evaluating mathematical reasoning in LLMs is constrained by limited benchmark sizes and inherent model stochasticity, yielding high-variance accuracy estimates and unstable rankings across platforms. On difficult problems, an LLM may fail…
Symbolic execution now becomes an indispensable technique for software testing and program analysis. There are several symbolic execution tools available off-the-shelf, and we need a practical benchmark approach to learn their capabilities.…
As experimental platforms for quantum information processing continue to mature, characterization of the quality of unitary gates that can be applied to their quantum bits (qubits) becomes essential. Eventually, the quality must be…
Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets. Benchmark prediction (also called efficient LLM evaluation) aims to select a small subset…
This paper describes our approach to automated program repair. We combine various techniques from the literature to achieve this. Our experiments show that our approach performs better than other techniques on standard benchmarks. However,…
Benchmarking is how the performance of a computing system is determined. Surprisingly, even for classical computers this is not a straightforward process. One must choose the appropriate benchmark and metrics to extract meaningful results.…
The behavior of real quantum hardware differs strongly from the simple error models typically used when simulating quantum error correction. Error processes are far more complex than simple depolarizing noise applied to single gates, and…
Accurate benchmarking of quantum gates is crucial for understanding and enhancing the performance of quantum hardware. A standard method for this is interleaved benchmarking, a technique which estimates the error on an interleaved target…
I would like to share recommendations on how to do performance benchmarks for the purpose of computer science research evaluation. Research in my field (programming language research) often involves performance considerations, but it is…
Elicited performance requirements need to be quantified for compliance in different engineering tasks, e.g., configuration tuning and performance testing. Much existing work has relied on manual quantification, which is expensive and…
Noise is typically treated as the adversary of quantum information processing. For open quantum dynamics, however, dissipation is part of the target physics, creating a tension with fault-tolerant architectures designed to suppress…
Numerous benchmarks aim to evaluate the capabilities of Large Language Models (LLMs) for causal inference and reasoning. However, many of them can likely be solved through the retrieval of domain knowledge, questioning whether they achieve…
We discuss guidelines for evaluating the performance of parameterized stochastic solvers for optimization problems, with particular attention to systems that employ novel hardware, such as digital quantum processors running variational…
Large language models (LLMs), a recent advance in deep learning and machine intelligence, have manifested astonishing capacities, now considered among the most promising for artificial general intelligence. With human-like capabilities,…
Issues concerning intelligent data analysis occurring in machine learning are investigated. A scheme for synthesizing correct supervised classification procedures is proposed. These procedures are focused on specifying partial order…
This position paper provides a critical but constructive discussion of current practices in benchmarking and evaluative practices in the field of formal reasoning and automated theorem proving. We take the position that open code, open…
A general error correction method is presented which is capable of correcting coherent errors originating from static residual inter-qubit couplings in a quantum computer. It is based on a randomization of static imperfections in a…
We describe the integration of logical and uncertain reasoning methods to identify the likely source and location of software problems. To date, software engineers have had few tools for identifying the sources of error in complex software…