Related papers: Performance evaluation through DEA benchmarking ad…

DevOps is a modern software engineering paradigm that is gaining widespread adoption in industry. The goal of DevOps is to bring software changes into production with a high frequency and fast feedback cycles. This conflicts with software…

Benchmarking tools, including stochastic frontier analysis (SFA), data envelopment analysis (DEA), and its stochastic extension (StoNED) are core tools in economics used to estimate an efficiency envelope and production inefficiencies from…

Dynamic optimization, for which the objective functions change over time, has attracted intensive investigations due to the inherent uncertainty associated with many real-world problems. For its robustness with respect to noise,…

We propose a novel DEA ranking based on a robust optimization viewpoint: the higher ranking for those DMU's that remain efficient even for larger variations of data and vice versa. This ranking can be computed by solving generalized linear…

I would like to share recommendations on how to do performance benchmarks for the purpose of computer science research evaluation. Research in my field (programming language research) often involves performance considerations, but it is…

Frequentist statistical methods, such as hypothesis testing, are standard practice in papers that provide benchmark comparisons. Unfortunately, these methods have often been misused, e.g., without testing for their statistical test…

Empirical and LLM-based research in model-driven engineering increasingly relies on datasets of software models, for instance, to train or evaluate machine learning techniques for modeling support. These datasets have a significant impact…

Benchmarking has long served as a foundational practice in machine learning and, increasingly, in modern AI systems such as large language models, where shared tasks, metrics, and leaderboards offer a common basis for measuring progress and…

The controller design of the so-called "difference algebraic equation" (DAE) systems that are frequently shown in industrial processes, tend to be challenging because of the combination of algebraic equations and high state dimensions. In…

Attribution maps for semantic segmentation are almost always judged by visual plausibility. Yet looking convincing does not guarantee that the highlighted pixels actually drive the model's prediction, nor that attribution credit stays…

Large language model-based agents show promise for software engineering, but environment configuration remains a bottleneck due to heavy manual effort and scarce large-scale, high-quality datasets. Existing benchmarks assess only end-to-end…

How do analysis goals and context affect exploratory data analysis (EDA)? To investigate this question, we conducted semi-structured interviews with 18 data analysts. We characterize common exploration goals: profiling (assessing data…

This paper proposes a new method to evaluate Decision Making Units (DMUs) under uncertainty using fuzzy Data Envelopment Analysis (DEA). In the proposed multi-objective nonlinear programming methodology both the objective functions and the…

Commonly, AI or machine learning (ML) models are evaluated on benchmark datasets. This practice supports innovative methodological research, but benchmark performance can be poorly correlated with performance in real-world applications -- a…

We introduce Meta Agents Research Environments (ARE), a research platform for scalable creation of environments, integration of synthetic or real applications, and execution of agentic orchestrations. ARE provides simple abstractions to…

Embedding-based entity alignment (EEA) has recently received great attention. Despite significant performance improvement, few efforts have been paid to facilitate understanding of EEA methods. Most existing studies rest on the assumption…

In this article, the concepts of technical efficiency, efficiency, effectiveness and productivity are illustrated. It is discussed that when firms are not homogenous, the situation is the same as when each factor has a different unit of…

Sufficient numbers of Decision Making Units (DMUs) in comparison with the number of input and output variables has been a concern of using Data Envelopment Analysis (DEA) in the last three decades. There are several studies in the…

Benchmarking functionalities in current commercial process mining tools allow organizations to contextualize their process performance through high-level performance indicators, such as completion rate or throughput time. However, they do…

Deep learning-based automated image analysis (DL-AIA) has been shown to outperform trained pathologists in tasks related to feature quantification. Related to these capacities the use of DL-AIA tools is currently extending from…