Related papers: When is it Better to Compare than to Score?
Companies survey their customers to measure their satisfaction levels with the company and its services. The received responses are crucial as they allow companies to assess their respective performances and find ways to make needed…
Reward modeling is crucial for aligning large language models with human preferences, yet current approaches lack a principled mathematical framework for leveraging ordinal preference data. When human annotators provide graded preferences…
Items in many datasets can be arranged to a natural order. Such orders are useful since they can provide new knowledge about the data and may ease further data exploration and visualization. Our goal in this paper is to define a…
Comparing alternatives in pairs is a well-known method of ranking creation. Experts are asked to perform a series of binary comparisons and then, using mathematical methods, the final ranking is prepared. As experts conduct the individual…
Stochastic orders are very useful tool to compare the lifetimes of two coherent systems. We show that, under certain conditions, a coherent system of used components performs better (worse) than a used coherent system with respect to…
In responding to rating questions, an individual may give answers either according to his/her knowledge/awareness or to his/her level of indecision/uncertainty, typically driven by a response style. As ignoring this dual behaviour may lead…
The performance of a number of different measures of nonlinearity in a time series is compared numerically. Their power to distinguish noisy chaotic data from linear stochastic surrogates is determined by Monte Carlo simulation for a number…
In certain complex optimization tasks, it becomes necessary to use multiple measures to characterize the performance of different algorithms. This paper presents a method that combines ordinal effect sizes with Pareto dominance to analyze…
Learning a reward model (RM) from human preferences has been an important component in aligning large language models (LLMs). The canonical setup of learning RMs from pairwise preference data is rooted in the classic Bradley-Terry (BT)…
Most popular strategies to capture subjective judgments from humans involve the construction of a unidimensional relative measurement scale, representing order preferences or judgments about a set of objects or conditions. This information…
We initiate the study of fairness for ordinal regression. We adapt two fairness notions previously considered in fair ranking and propose a strategy for training a predictor that is approximately fair according to either notion. Our…
Our goal is a mechanism for efficiently assigning scalar ratings to each of a large set of elements. For example, "what percent positive or negative is this product review?" When sample sizes are small, prior work has advocated for methods…
There is some disagreement on whether Likert scale data should be treated as ordinal or continuous. This paper treats Likert data as ordinal, uses non-parametric hypothesis testing, and clustering to validate those variables that have…
Modern online platforms rely on effective rating systems to learn about items. We consider the optimal design of rating systems that collect binary feedback after transactions. We make three contributions. First, we formalize the…
Ordinal data is widely prevalent in clinical and other domains, yet there is a lack of both modern, machine-learning based methods and publicly available software to address it. In this paper, we present a model-agnostic method of ordinal…
An ordinal classification problem is one in which the target variable takes values on an ordinal scale. Nowadays, there are many of these problems associated with real-world tasks where it is crucial to accurately classify the extreme…
This study introduces the "Grade Score", a novel metric designed to evaluate the consistency and fairness of Large Language Models (LLMs) when used as multiple-choice judges with respect to order bias and choice consistency. The Grade Score…
Ordinal peer grading has been proposed as a simple and scalable solution for computing reliable information about student performance in massive open online courses. The idea is to outsource the grading task to the students themselves as…
Many outcomes of interest in the social and health sciences, as well as in modern applications in computational social science and experimentation on social media platforms, are ordinal and do not have a meaningful scale. Causal analyses…
The goal of Ordinal Regression is to find a rule that ranks items from a given set. Several learning algorithms to solve this prediction problem build an ensemble of binary classifiers. Ranking by Projecting uses interdependent binary…