Related papers: Notes on the H-measure of classifier performance
While advanced classifiers have been increasingly used in real-world safety-critical applications, how to properly evaluate the black-box models given specific human values remains a concern in the community. Such human values include…
The evaluation of a researcher's performance has traditionally relied on various bibliometric measures, with the h-index being one of the most prominent. However, the h-index only accounts for the number of citations received in a…
In this paper, the authors propose a software metric called Class Activeness Metric which helps to determine the level of accessibility of the members of a class when it is instantiated as objects. Object interactions need to be straight…
Quantification is the machine learning task of estimating test-data class proportions that are not necessarily similar to those in training. Apart from its intrinsic value as an aggregate statistic, quantification output can also be used to…
Data quality is a key element for building and optimizing good learning models. Despite many attempts to characterize data quality, there is still a need for rigorous formalization and an efficient measure of the quality from available…
The majority of automatic metrics for evaluating NLG systems are reference-based. However, the challenge of collecting human annotation results in a lack of reliable references in numerous application scenarios. Despite recent advancements…
When learning a new concept, not all training examples may prove equally useful for training: some may have higher or lower training value than others. The goal of this paper is to bring to the attention of the vision community the…
How can one meaningfully make a measurement, if the meter does not conform to any standard and its scale expands or shrinks depending on what is measured? In the present work it is argued that current evaluation practices for…
Quantile-based classifiers can classify high-dimensional observations by minimising a discrepancy of an observation to a class based on suitable quantiles of the within-class distributions, corresponding to a unique percentage for all…
The complex software systems developed nowadays require assessing their quality and proneness to errors. Reducing code complexity is a never-ending problem, especially in today's fast pace of software systems development. Therefore, the…
Capital allocation principles are used in various contexts in which a risk capital or a cost of an aggregate position has to be allocated among its constituent parts. We study capital allocation principles in a performance measurement…
Machine learning (ML) is the field of training machines to achieve high level of cognition and perform human-like analysis. Since ML is a data-driven approach, it seemingly fits into our daily lives and operations as well as complex and…
In this note, we present a novel measure of similarity between two functions. It quantifies how the sub-optimality gaps of two functions convert to each other, and unifies several existing notions of functional similarity. We show that it…
Nearly a decade ago, the science community was introduced to the $h$-index, a proposed statistical measure of the collective impact of the publications of any individual researcher. It is of course undeniable that any method of reducing a…
The predictive power of the h-index has been shown to depend for a long time on citations to rather old publications. This has raised doubts about its usefulness for predicting future scientific achievements. Here I investigate a variant…
We develop a new approach to solving classification problems, which is bases on the theory of coherent measures of risk and risk sharing ideas. The proposed approach aims at designing a risk-averse classifier. The new approach allows for…
A variety of different performance metrics are commonly used in the machine learning literature for the evaluation of classification systems. Some of the most common ones for measuring quality of hard decisions are standard and balanced…
The robustness of classifiers has become a question of paramount importance in the past few years. Indeed, it has been shown that state-of-the-art deep learning architectures can easily be fooled with imperceptible changes to their inputs.…
Learning classifiers using skewed or imbalanced datasets can occasionally lead to classification issues; this is a serious issue. In some cases, one class contains the majority of examples while the other, which is frequently the more…
Classification tasks are common across many fields and applications where the decision maker's action is limited by resource constraints. In direct marketing only a subset of customers is contacted; scarce human resources limit the number…