Related papers: AWAKE-related benchmarking tests for simulation co…
In this paper in which we address the evaluation of measurement process quality, we mainly focus on the evaluation procedure, as far as it is based on the numerical measurement outcomes. We challenge the approach where the "exact" value of…
Software testing is a critical aspect of software development, yet generating test cases remains a routine task for engineers. This paper presents a benchmark, CLOVER, to evaluate models' capabilities in generating and completing test cases…
A recent case study from AWS by Chong et al. proposes an effective methodology for Bounded Model Checking in industry. In this paper, we report on a follow up case study that explores the methodology from the perspective of three research…
Learners are often introduced to programming via dedicated languages such as Scratch, where block-based commands are assembled visually in order to control the interactions of graphical sprites. Automated testing of such programs is an…
Addressing the reproducibility crisis in artificial intelligence through the validation of reported experimental results is a challenging task. It necessitates either the reimplementation of techniques or a meticulous assessment of papers…
In operating system development, concurrency poses significant challenges. It is difficult for humans to manually review concurrent behaviors or to write test cases covering all possible executions, often resulting in critical bugs.…
An application's performance regressions can be detected by both application or microbenchmarks. While application benchmarks stress the system under test by sending synthetic but realistic requests which, e.g., simulate real user traffic,…
Now that ever more sophisticated devices for quantum computing are being developed, we require ever more sophisticated benchmarks. This includes a need to determine how well these devices support the techniques required for quantum error…
Recent advances in Large Language Models (LLMs) have demonstrated strong potential in code generation, yet their effectiveness in quantum computing remains underexplored. This paper benchmarks LLMs for PennyLane-based quantum code…
AlphaCode is a code generation system for assisting software developers in solving competitive programming problems using natural language problem descriptions. Despite the advantages of the code generating system, the open source community…
Concomitant with the rapid development of quantum technologies, challenging demands arise concerning the certification and characterization of devices. The promises of the field can only be achieved if stringent levels of precision of…
Equivalence checking of quantum circuits is an essential element in quantum program compilation, in which a quantum program can be synthesized into different quantum circuits that may vary in the number of qubits, initialization…
We consider statistical methods based on finite samples of locally randomized measurements in order to certify different degrees of multiparticle entanglement in intermediate-scale quantum systems. We first introduce hierarchies of…
Quantum processors are now able to run quantum circuits that are infeasible to simulate classically, creating a need for benchmarks that assess a quantum processor's rate of errors when running these circuits. Here, we introduce a general…
Three numerical coverage metrics for the symbolic simulation of dense-time systems and their estimation methods are presented. Special techniques to derive numerical estimations of dense-time state-spaces have also been developed.…
Evaluating Software testability can assist software managers in optimizing testing budgets and identifying opportunities for refactoring. In this paper, we abandon the traditional approach of pursuing testability measurements based on the…
Novel reinforcement learning algorithms, or improvements on existing ones, are commonly justified by evaluating their performance on benchmark environments and are compared to an ever-changing set of standard algorithms. However, despite…
This report describes a variety of programming assignments that can be used to teach quantum computing in a practical manner. These assignments let the learners get hands-on experience with all stages of quantum software development…
In this work, we present a new benchmarking suite with new real-life inspired skewed workloads to test the performance of concurrent index data structures. We started this project to prepare workloads specifically for self-adjusting data…
Cross-coverage of a program P refers to the test coverage measured over a different program Q that is functionally equivalent to P. The novel concept of cross-coverage can find useful applications in the test of redundant software. We apply…