Related papers: WURBench: Toward Benchmarking Wake-up Radio-based …
Benchmarking involves designing, running and disseminating rigorous performance assessments of methods, most often for data analysis and software tools, but the process can also be applied to experimental systems. Ideally, a benchmarking…
The quest for precision in parameter estimation is a fundamental task in different scientific areas. The relevance of this problem thus provided the motivation to develop methods for the application of quantum resources to estimation…
As artificial intelligence systems grow more powerful, there has been increasing interest in "AI safety" research to address emerging and future risks. However, the field of AI safety remains poorly defined and inconsistently measured,…
Because of increasing amounts of intermittent and distributed generators in power systems, many demand response programs have been developed to schedule flexible energy consumption. However, proper benchmarks for comparing these methods are…
In safety-critical deep learning applications, robustness measures the ability of neural models that handle imperceptible perturbations in input data, which may lead to potential safety hazards. Existing pre-deployment robustness assessment…
Predictive benchmarking, the evaluation of machine learning models based on predictive performance and competitive ranking, is a central epistemic practice in machine learning research and an increasingly prominent method for scientific…
Performance evaluation is a key issue for designers and users of Database Management Systems (DBMSs). Performance is generally assessed with software benchmarks that help, e.g., test architectural choices, compare different technologies or…
Performance benchmarking is a common practice in software engineering, particularly when building large-scale, distributed, and data-intensive systems. While cloud environments offer several advantages for running benchmarks, it is often…
The increasing attention on deep learning has tremendously spurred the design of intelligence processing hardware. The variety of emerging intelligence processors requires standard benchmarks for fair comparison and system optimization (in…
Subjective listening tests remain the golden standard for speech quality assessment, but are costly, variable, and difficult to scale. In contrast, existing objective metrics, such as PESQ, F0 correlation, and DNSMOS, typically capture only…
Performing an overview of the benchmarking initiatives oriented towards the performance evaluation of Holonic Manufacturing Systems shows that there are very few of them. However, a comparison between all the isolated emu-lation…
Stationary collectors reading wireless, battery powered smart meters, often operate in harsh channel conditions to cut network installation cost to a minimum, challenging the individual link to each meter. The desired performance measure is…
Signal-to-noise ratios are a widely used concept for astroparticle radio detectors, such as air-shower radio arrays for cosmic-ray measurements or detectors searching for radio signals induced by neutrino interactions in ice. Nonetheless,…
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges…
Sleep is among the most important factors affecting one's daily performance, well-being, and life quality. Nevertheless, it became possible to measure it in daily life in an unobtrusive manner with wearable devices. Rather than camera…
Despite significant progress in designing powerful adversarial evasion attacks for robustness verification, the evaluation of these methods often remains inconsistent and unreliable. Many assessments rely on mismatched models, unverified…
Wireless sensing has been recognized as a key enabling technology for numerous emerging applications. For decades, the sensing performance was mostly evaluated from a reliability perspective, with the efficiency aspect widely unexplored.…
Wireless systems are getting deployed in many new environments with different antenna heights, frequency bands and multipath conditions. This has led to an increasing demand for more channel measurements to understand wireless propagation…
The absence of a common framework for benchmarking quantum networks is an obstacle to comparing the capabilities of different quantum networks. We propose a general framework for quantifying the performance of a quantum network, which is…
While deep learning methods continue to improve in predictive accuracy on a wide range of application domains, significant issues remain with other aspects of their performance including their ability to quantify uncertainty and their…