Related papers: An Axiom System for Feedback Centralities
The quality of a summarization evaluation metric is quantified by calculating the correlation between its scores and human annotations across a large number of summaries. Currently, it is unclear how precise these correlation estimates are,…
In many classification tasks, there is no definitive ground truth, only human judgments that may disagree. We address two challenges that arise in such settings: (1) how to use human raters to score classifiers, and (2) how to use them for…
Concept explanation is a popular approach for examining how human-interpretable concepts impact the predictions of a model. However, most existing methods for concept explanations are tailored to specific models. To address this issue, this…
The axiomatic foundations of Bentham and Rawls solutions are discussed within the broader domain of cardinal preferences. It is unveiled that both solution concepts share all four of the following axioms: Nonemptiness, Anonymity, Unanimity,…
Spectral centrality measures allow to identify influential individuals in social groups, to rank Web pages by their popularity, and even to determine the impact of scientific researches. The centrality score of a node within a network…
To overcome the limitations of automated metrics (e.g. BLEU, METEOR) for evaluating dialogue systems, researchers typically use human judgments to provide convergent evidence. While it has been demonstrated that human judgments can suffer…
There is no single universally accepted definition of "Complexity". There are several perspectives on complexity and what constitutes complex behaviour or complex systems, as opposed to regular, predictable behaviour and simple systems. In…
Modern language models often exhibit powerful but brittle behavior, leading to the development of larger and more diverse benchmarks to reliably assess their behavior. Here, we suggest that model performance can be benchmarked and…
This paper presents a novel quantitative approach for comparative economic studies, addressing limitations in current classification methods. Conventional approaches in comparative economics often rely on ad hoc and categorical…
People use the world wide web heavily to share their experience with entities such as products, services, or travel destinations. Texts that provide online feedback in the form of reviews and comments are essential to make consumer…
In the realm of artificial intelligence, where a vast majority of data is unstructured, obtaining substantial amounts of labeled data to train supervised machine learning models poses a significant challenge. To address this, we delve into…
Contextuality is a central property in comparative analysis of classical, quantum, and supercorrelated systems. We examine and compare two well-motivated approaches to contextuality. One approach ("contextuality-by-default") is based on the…
Evaluating statement autoformalization, translating natural language mathematics into formal languages like Lean 4, remains a significant challenge, with few metrics, datasets, and standards to robustly measure progress. In this work, we…
Generative AI systems have become ubiquitous for all kinds of modalities, which makes the issue of the evaluation of such models more pressing. One popular approach is preference ratings, where the generated outputs of different systems are…
The importance of a node in a social network is identified through a set of measures called centrality. Degree centrality, closeness centrality, betweenness centrality and clustering coefficient are the most frequently used metrics to…
As a key method in dealing with uncertainties, feedback has been understood fairly well in classical control theory. But for quantum control systems, the capability of measurement-based feedback control (MFC) has not been investigated…
Automatic systems are increasingly used to assess the originality of responses in creative tasks. They offer a potential solution to key limitations of human assessment (cost, fatigue, and subjectivity), but there is preliminary evidence of…
We take an axiomatic approach to study redistribution problems when agents report income and needs. We formalize axioms reflecting ethical and operational principles such as additivity, impartiality and individual rationality. Different…
Validation plays a crucial role in the clustering process. Many different internal validity indexes exist for the purpose of determining the best clustering solution(s) from a given collection of candidates, e.g., as produced by different…
For various speech-related tasks, confidence scores from a speech recogniser are a useful measure to assess the quality of transcriptions. In traditional hidden Markov model-based automatic speech recognition (ASR) systems, confidence…