Related papers: A new metric of consensus for Likert scales

A ranking is an ordered sequence of items, in which an item with higher ranking score is more preferred than the items with lower ranking scores. In many information systems, rankings are widely used to represent the preferences over a set…

Many organizations describe their processes as consensus-driven, but there is no consensus on the definition of consensus. Qualitative definitions of consensus prioritize social phenomena like "unity" that are not necessarily measurable.…

The comparison of alternative rankings of a set of items is a general and prominent task in applied statistics. Predictor variables are ranked according to magnitude of association with an outcome, prediction models rank subjects according…

In this technical note, we introduce a novel approach to studying consensus of continuous-time nonlinear systems with varying topology based on Hilbert metric. We demonstrate that this metric offers significant flexibility in analyzing…

This work is motivated by the need to assess the degree of agreement between two independent groups of raters. It proposes two new methods.

Rankings, representing preferences over a set of candidates, are widely used in many information systems, e.g., group decision making and information retrieval. It is of great importance to evaluate the consensus of the obtained rankings…

This paper proposes a new distance metric between clusterings that incorporates information about the spatial distribution of points and clusters. Our approach builds on the idea of a Hilbert space-based representation of clusters as a…

This paper is concerned with the probability of consensus in a multivariate, spatially explicit version of the Hegselmann-Krause model for the dynamics of opinions. Individuals are located on the vertices of a finite connected graph…

This paper proposes the matrix-weighted consensus algorithm, which is a generalization of the consensus algorithm in the literature. Given a networked dynamical system where the interconnections between agents are weighted by nonnegative…

This paper primarily demonstrates a method to quantitatively assess the alignment between multi-step, structured reasoning in large language models and human preferences. We introduce the Alignment Score, a semantic-level metric that…

A consensus system is a linear multi-agent system in which agents communicate to reach a so-called consensus state, defined as the average of the initial states of the agents. Consider a more generalized situation in which each agent is…

This paper introduces the concept of {mutual consensus} as a novel non-compensatory consensus measure that accounts for the maximum disparity among opinions to ensure robust consensus evaluation. Incorporating this concept, several new…

Recent work has sought to quantify large language model uncertainty to facilitate model control and modulate user trust. Previous works focus on measures of uncertainty that are theoretically grounded or reflect the average overt behavior…

Existing metrics in competing risks survival analysis such as concordance and accuracy do not evaluate a model's ability to jointly predict the event type and the event time. To address these limitations, we propose a new metric, which we…

Ordered response scales are ubiquitous in economics, but their interpretation rests on an untested assumption: that numerical labels reflect equal psychological intervals. The contribution of this paper is to provide a systematic assessment…

A suitable scalar metric can help measure multi-calibration, defined as follows. When the expected values of observed responses are equal to corresponding predicted probabilities, the probabilistic predictions are known as "perfectly…

Opinion polls remain among the most efficient and widespread methods to capture psycho-social data at large scales. However, there are limitations on the logistics and structure of opinion polls that restrict the amount and type of…

This paper evaluates the dynamic response of economic activity to shocks in uncertainty as percieved by agents.The study focuses on the comparison between the perception of economic uncertainty by manufacturers and consumers.Since…

A primary goal of online deliberation platforms is to identify ideas that are broadly agreeable to a community of users through their expressed preferences. Yet, consensus elicitation should ideally extend beyond the specific statements…

We analyze the structure of the disagreement among a population of voters over a set of alternatives. Surveys typically ask either for pairwise comparisons, simple and intuitive for participants, or full rankings over alternatives,…