Related papers: WURBench: Toward Benchmarking Wake-up Radio-based …
Reinforcement learning has recently experienced increased prominence in the machine learning community. There are many approaches to solving reinforcement learning problems with new techniques developed constantly. When solving problems…
Noise power estimation is a key issue in modern wireless communication systems. It allows resource allocation by detecting white spectral spaces effectively, and gives control over the communication process by adjusting transmission power.…
In the near-term "NISQ"-era of noisy, intermediate-scale, quantum hardware and beyond, reliably determining the quality of quantum devices becomes increasingly important: users need to be able to compare them with one another, and make an…
Ranked lists are frequently used by information retrieval (IR) systems to present results believed to be relevant to the users information need. Fairness is a relatively new but important aspect of these rankings to measure, joining a rich…
Our research aims to propose a new performance-explainability analytical framework to assess and benchmark machine learning methods. The framework details a set of characteristics that systematize the performance-explainability assessment…
Quantum communication devices, such as quantum repeaters, quantum memories, or quantum channels, are unavoidably exposed to imperfections. However, the presence of imperfections can be tolerated, as long as we can verify such devices retain…
Big data systems address the challenges of capturing, storing, managing, analyzing, and visualizing big data. Within this context, developing benchmarks to evaluate and compare big data systems has become an active topic for both research…
Nanopore sequencing technologies continue to advance rapidly, offering critical benefits such as real-time analysis, the ability to sequence extremely long DNA fragments (up to millions of bases in a single read), and the option to…
Over 80% of wireless traffic already takes place in buildings. Like water, gas, and electricity, wireless communication is becoming one of the most fundamental utilities of a building. It is well known that building structures have a…
Many studies have shown automatic speech processing (ASR) systems have unequal performance across speakergroups (SG's). However, the manner in which such studies arrive at this conclusion is inconsistent. To pave the wayfor more reliable…
The rapid pace of development in quantum computing technology has sparked a proliferation of benchmarks for assessing the performance of quantum computing hardware and software. Good benchmarks empower scientists, engineers, programmers,…
Trustworthiness assessment is an essential step to assure that interdependent systems perform critical functions as anticipated, even under adverse conditions. In this paper, a holistic trustworthiness assessment framework for…
Quantum optimisation is emerging as a promising approach alongside classical heuristics and specialised hardware, yet its performance is often difficult to assess fairly. Traditional benchmarking methods, rooted in digital complexity…
Wake-Up Radio (WUR) enables resource-constrained, battery-powered sensor nodes to remain in a low-power deep sleep state while continuously listening for a Wake-Up Signal (WUS). Sensor nodes only wake and transmit data after receiving the…
The rapid advancement of fake voice generation technology has ignited a race with detection systems, creating an urgent need to secure the audio ecosystem. However, existing benchmarks suffer from a critical limitation: they typically…
Benchmarking is crucial for testing and validating any system, even more so in real-time systems. Typical real-time applications adhere to well-understood abstractions: they exhibit a periodic behavior, operate on a well-defined working…
Emerging wireless technologies with Gbps connectivity, such as the 5th generation (5G) and 6th generation (6G) of mobile networks, require improved and substantiating documentation for the wireless standards concerning the radio signals,…
Neural information retrieval (IR) systems have progressed rapidly in recent years, in large part due to the release of publicly available benchmarking tasks. Unfortunately, some dimensions of this progress are illusory: the majority of the…
Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-based systems, but developers nonetheless claim that their…
I would like to share recommendations on how to do performance benchmarks for the purpose of computer science research evaluation. Research in my field (programming language research) often involves performance considerations, but it is…