Related papers: Measuring intergroup agreement and disagreement
Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged…
A distinction has been drawn in fair machine learning research between `group' and `individual' fairness measures. Many technical research papers assume that both are important, but conflicting, and propose ways to minimise the trade-offs…
Pairwise comparisons are an important tool of modern (multiple criteria) decision making. Since human judgments are often inconsistent, many studies focused on the ways how to express and measure this inconsistency, and several…
Networks, representing attitudinal survey data, expose the structure of opinion-based groups. We make use of these network projections to identify the groups reliably through community detection algorithms and to examine…
The growing need for trustworthy machine learning has led to the blossom of interpretability research. Numerous explanation methods have been developed to serve this purpose. However, these methods are deficiently and inappropriately…
We study the merging and the testing of opinions in the context of a prediction model. In the absence of incentive problems, opinions can be tested and rejected, regardless of whether or not data produces consensus among Bayesian agents. In…
Increasingly, critical decisions in public policy, governance, and business strategy rely on a deeper understanding of the needs and opinions of constituent members (e.g. citizens, shareholders). While it has become easier to collect a…
In this paper peer review reliability is investigated based on peer ratings of research teams at two Belgian universities. It is found that outcomes can be substantially influenced by the different ways in which experts attribute ratings.…
Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information…
Because high-quality data is like oxygen for AI systems, effectively eliciting information from crowdsourcing workers has become a first-order problem for developing high-performance machine learning algorithms. Two prevalent paradigms,…
The manuscript is about an interlaboratory comparison which involved eleven metrology institutes. It comprises four tasks: i) deriving a consensus value from these results; ii) evaluating the associated standard uncertainty; iii) producing…
How to robustly rank the aesthetic quality of given images has been a long-standing ill-posed topic. Such challenge stems mainly from the diverse subjective opinions of different observers about the varied types of content. There is a…
Quantifying cooperation or synergy among random variables in predicting a single target random variable is an important problem in many complex systems. We review three prior information-theoretic measures of synergy and introduce a novel…
Ingroup favoritism, the tendency to favor ingroup over outgroup, is often explained as a product of intergroup conflict, or correlations between group tags and behavior. Such accounts assume that group membership is meaningful, whereas…
In unsupervised machine learning, agreement between partitions is commonly assessed with so-called external validity indices. Researchers tend to use and report indices that quantify agreement between two partitions for all clusters…
Studies on interpersonal conflict have a long history and contain many suggestions for conflict typology. We use this as the basis of a novel annotation scheme and release a new dataset of situations and conflict aspect annotations. We then…
We propose, develop, and analyze a mathematical model of intergroup attitudes in a community that is divided between two distinct social groups (which may be distinguished by religion, ethnicity, or some other socially distinguishing…
Agreement between measurement methods is commonly investigated by a so-called Bland-Altman plot showing if the difference is independent of the size of the measurement. However, such analysis assumes that both methods have the same…
We explore a new mechanism to explain polarization phenomena in opinion dynamics in which agents evaluate alternative views on the basis of the social feedback obtained on expressing them. High support of the favored opinion in the social…
In many cases, recommendations are consumed by groups of users rather than individuals. In this paper, we present a system which recommends social events to groups. The system helps groups to organize a joint activity and collectively…