Related papers: How important tasks are performed: peer review
Feedback has a powerful influence on learning, but it is also expensive to provide. In large classes, it may even be impossible for instructors to provide individualized feedback. Peer assessment has received attention lately as a way of…
We explore how the mechanics of collective decision-making, especially of jury deliberation, can be inferred from macroscopic statistics. We first hypothesize that the dynamics of competing opinions can leave a "fingerprint" in the joint…
Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1) providing an opportunity to study this important…
Scientists pursue collective knowledge, but they also seek personal recognition from their peers. When scientists decide whether or not to work on a big new problem, they weigh the potential rewards of a major discovery against the costs of…
When do we stop an ongoing science project to make room for something new? Decision-making is a complex process, ranging from budgetary considerations and tension between ongoing projects, to progress assessments and allowance for novel…
Understanding the relationship between the composition of a research team and the potential impact of their research papers is crucial as it can steer the development of new science policies for improving the research enterprise. Numerous…
A toy-model of publications and citations processes is proposed. The model shows that the role of randomness in the processes is essential and cannot be ignored. Some other aspects of scientific publications rating are discussed.
The last few years have seen the proliferation of measures that quantify the scientific output of researches. Yet, these measures focus on productivity, thus fostering the "publish or perish" paradigm. This article proposes a measure that…
Peer review is a core element of the scientific process, particularly in conference-centered fields such as ML and NLP. However, only few studies have evaluated its properties empirically. Aiming to fill this gap, we present a corpus that…
A semi-supervised model of peer review is introduced that is intended to overcome the bias and incompleteness of traditional peer review. Traditional approaches are reliant on human biases, while consensus decision-making is constrained by…
Large parts of professional human communication proceed in a request-reply fashion, whereby requests contain specifics of the information desired while replies can deliver the required information. However, time limitations often force…
This paper presents a novel approach to analyze human decision-making that involves comparing the behavior of professional chess players relative to a computational benchmark of cognitively bounded rationality. This benchmark is constructed…
Peer review is the method employed by the scientific community for evaluating research advancements. In the field of cybersecurity, the practice of double-blind peer review is the de-facto standard. This paper touches on the holy grail of…
In seeking to understand the processes enacted during software development, an increasing number of studies have mined software repositories. In particular, studies have endeavored to show how teams resolve software defects. Although much…
Most people participate in meetings almost every day, multiple times a day. The study of meetings is important, but also challenging, as it requires an understanding of social signals and complex interpersonal dynamics. Our aim this work is…
We expect that democracy enables us to utilize collective intelligence such that our collective decisions build and enhance social welfare, and such that we accept their distributive and normative consequences. Collective decisions are…
The computing research community needs to work much harder to address the downsides of our innovations. Between the erosion of privacy, threats to democracy, and automation's effect on employment (among many other issues), we can no longer…
The peer review process is often regarded as the gatekeeper of scientific integrity, yet increasing evidence suggests that it is not immune to bias. Although structural inequities in peer review have been widely debated, much less attention…
A continuous-time Markov process is proposed to analyze how a group of humans solves a complex task, consisting in the search of the optimal set of decisions on a fitness landscape. Individuals change their opinions driven by two different…
Manuscripts have a complex development process with multiple influencing factors. Reconstructing this process is difficult without large-scale, comparable data on different versions of manuscripts. Preprints are increasingly available and…