Related papers: Evaluation for Change
Recent years have witnessed an increasing number of interpretation methods being developed for improving transparency of NLP models. Meanwhile, researchers also try to answer the question that whether the obtained interpretation is faithful…
NLP researchers regularly invoke abstract concepts like "interpretability," "bias," "reasoning," and "stereotypes," without defining them. Each subfield has a shared understanding or conceptualization of what these terms mean and how we…
While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural…
We survey a range of models of opinion exchange. From the introduction: "The exchange of opinions between individuals is a fundamental social interaction... Moreover, many models in this field are an excellent playground for mathematicians,…
Opinions are an integral part of how we perceive the world and each other. They shape collective action, playing a role in democratic processes, the evolution of norms, and cultural change. For decades, researchers in the social and natural…
Opinion evolution mechanism can be captured by physics modeling approaches. In this context, a kinetic equation is established by defining a generalized displace (cognitive-level), a driving force and the related generalized potential,…
The manifestation and effect of bias in news reporting have been central topics in the social sciences for decades, and have received increasing attention in the NLP community recently. While NLP can help to scale up analyses or contribute…
We formalize and study a phenomenon called feature collapse that makes precise the intuitive idea that entities playing a similar role in a learning task receive similar representations. As feature collapse requires a notion of task, we…
Equal opportunity is central to the concept of meritocracy. Opportunity and leadership should go to the people most qualified by performance, and not on the basis of arbitrary or irrelevant attributes. This principle is arguably most…
This paper explores the topic of transportability, as a sub-area of generalisability. By proposing the utilisation of metrics based on well-established statistics, we are able to estimate the change in performance of NLP models in new…
Opinion dynamics of a group of individuals is the change in the members' opinions through mutual interaction with each other. The related literature contains works in which the dynamics is modeled as a continuous system, of which behavioral…
State-of-the-art models in NLP are now predominantly based on deep neural networks that are opaque in terms of how they come to make predictions. This limitation has increased interest in designing more interpretable deep models for NLP…
There are many on-line settings in which users publicly express opinions. A number of these offer mechanisms for other users to evaluate these opinions; a canonical example is Amazon.com, where reviews come with annotations like "26 of 32…
The objective of this paper is to design performance metrics and respective formulas to quantitatively evaluate the achievement of set objectives and expected outcomes both at the course and program levels. Evaluation is defined as one or…
Natural language processing for programming aims to use NLP techniques to assist programming. It is increasingly prevalent for its effectiveness in improving productivity. Distinct from natural language, a programming language is highly…
Communication plays a pivotal role in driving change, influencing human behavior, and solving problems. Chapter 2 explores the pragmatic power of communication, drawing examples and research to illustrate its impact on people's perceptions…
In Natural Language Processing (NLP), the Elo rating system, originally designed for ranking players in dynamic games such as chess, is increasingly being used to evaluate Large Language Models (LLMs) through "A vs B" paired comparisons.…
Recent advancements in NLP systems, particularly with the introduction of LLMs, have led to widespread adoption of these systems by a broad spectrum of users across various domains, impacting decision-making, the job market, society, and…
Modern deep learning models for NLP are notoriously opaque. This has motivated the development of methods for interpreting such models, e.g., via gradient-based saliency maps or the visualization of attention weights. Such approaches aim to…
Usability is often defined as the ability of a system to carry out specific tasks by specific users in a specific context. Usability evaluation involves testing the system for its expected usability. Usability testing is performed in…