Related papers: Cross-benchmarking for performance evaluation: loo…
Benchmarking, which involves collecting reference datasets and demonstrating method performances, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of…
This article presents a Benchmarking methodology to support decision-making for international market selection (IMS). In order to do so, we will be using an output-oriented Data Envelopment Analysis (DEA) model. This methodology considers…
An alternative approach for the panel second stage of data envelopment analysis (DEA) is presented in this paper. Instead of efficiency scores, we propose to model rankings in the second stage using a dynamic ranking model in the…
The need for performance measurement tools appeared soon after the emergence of the first Object-Oriented Database Management Systems (OODBMSs), and proved important for both designers and users (Atkinson \& Maier, 1990). Performance…
Mixture-of-Experts (MoE) architectures have emerged as a promising direction, offering efficiency and scalability by activating only a subset of parameters during inference. However, current research remains largely performance-centric,…
The objective of this paper is to evaluate the performance of decision-making units (DMUs) using a hybrid fuzzy multi-objective (FMO) data envelopment analysis (DEA) approach. This study develops fuzzy multi-objective optimistic (FMOO) and…
Modern methods for multi-criteria assessment (MCA), such as Data Envelopment Analysis (DEA), Stochastic Frontier Analysis (SFA), and Multiple Criteria Decision-Making (MCDM), are utilized to appraise a collection of Decision-Making Units…
In the context of cluster analysis and graph partitioning, many external evaluation measures have been proposed in the literature to compare two partitions of the same set. This makes the task of selecting the most appropriate measure for a…
Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape usability as much as model quality. Through a structured analysis of existing benchmark…
In Machine Learning, a benchmark refers to an ensemble of datasets associated with one or multiple metrics together with a way to aggregate different systems performances. They are instrumental in (i) assessing the progress of new methods…
Evaluating the banks' performance has always been of interest due to their crucial role in the economic development of each country. Data envelopment analysis (DEA) has been widely used for measuring the performance of bank branches. In the…
Robust benchmarks are crucial for evaluating Multimodal Large Language Models (MLLMs). Yet we find that models can ace many multimodal benchmarks without strong visual understanding, instead exploiting biases, linguistic priors, and…
In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the performance of different…
A `state of the art' model A surpasses humans in a benchmark B, but fails on similar benchmarks C, D, and E. What does B have that the other benchmarks do not? Recent research provides the answer: spurious bias. However, developing A to…
Benchmarking has long served as a foundational practice in machine learning and, increasingly, in modern AI systems such as large language models, where shared tasks, metrics, and leaderboards offer a common basis for measuring progress and…
AI models are increasingly prevalent in high-stakes environments, necessitating thorough assessment of their capabilities and risks. Benchmarks are popular for measuring these attributes and for comparing model performance, tracking…
This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses…
Assessing the technical efficiency of a set of observations requires that the associated data composed of inputs and outputs are perfectly known. If this is not the case, then biased estimates will likely be obtained. Data Envelopment…
Natural language models are often summarized through a high-dimensional set of descriptive metrics including training corpus size, training time, the number of trainable parameters, inference times, and evaluation statistics that assess…
Data Envelopment Analysis (DEA) appears more than just an instrument of measurement. DEA models can be seen as a mathematical structure for democratic voicing within decisional contexts. Such an important aspect of DEA is enhanced through…