Related papers: Algorithmic Evaluation and the Marginalization of …
As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e.g. training data), are deployed by multiple decision-makers. While sharing offers…
Ranking systems produce ordered lists from scalar scores, yet the ranking itself depends only on pairwise comparisons. We develop a mathematical theory that takes this observation seriously, centering the analysis on pairwise margins rather…
The lack of a macro-level, systematic evaluation theory to guide the implementation of evaluation practices has become a key bottleneck in the reform of global research evaluation systems. By reviewing the historical development of research…
Large language models (LLMs) are playing an increasingly integral, though largely informal, role in scholarly peer review. Yet it remains unclear whether LLMs reproduce the biases observed in human decision-making. We adapt a resume-style…
Meritocratic systems, from admissions to hiring, aim to impartially reward skill and effort. Yet persistent disparities across race, gender, and class challenge this ideal. Some attribute these gaps to structural inequality; others to…
Double-blind peer review is considered a pillar of academic research because it is perceived to ensure a fair, unbiased, and fact-centered scientific discussion. Yet, experienced researchers can often correctly guess from which research…
Peer review remains the central quality-control mechanism of science, yet its ability to fulfill this role is increasingly strained. Empirical studies document serious shortcomings: long publication delays, escalating reviewer burden…
Scientific progress has long been understood as recombinant, with breakthroughs arising when existing ideas are joined in new ways. Empirical work in this tradition has focused on the inputs to discovery, asking whether a paper draws…
International collaboration is sometimes encouraged in the belief that it generates higher quality research or is more capable of addressing societal problems. Nevertheless, while there is evidence that the journal articles of international…
Citation metrics are becoming pervasive in the quantitative evaluation of scholars, journals and institutions. More then ever before, hiring, promotion, and funding decisions rely on a variety of impact metrics that cannot disentangle…
Some research institutions demand researchers to distribute the incomes they earn from publishing papers to their researchers and/or co-authors. In this study, we deal with the Impact Factor-based ranking journal as a criteria for the…
Science has become more collaborative over the past years, a phenomenon that is related to the increase in the number of authors per paper and the emergence of interdisciplinary works featuring specialists of different fields. In such a…
Decentralization is a pervasive concept found across disciplines, including Economics, Political Science, and Computer Science, where it is used in distinct yet interrelated ways. Here, we develop and publicly release a general pipeline to…
Collectively, machine learning (ML) researchers are engaged in the creation and dissemination of knowledge about data-driven algorithms. In a given paper, researchers might aspire to any subset of the following goals, among others: to…
Several studies exist which use scientific literature for comparing scientific activities (e.g., productivity, and collaboration). In this study, using co-authorship data over the last 40 years, we present the evolutionary dynamics of multi…
Competent-looking judgment, including selecting, ranking, attributing, and certifying, is now produced at scale at marginal cost approaching zero, inverting the dominant economics-of-AI reading that treats judgment as the scarce complement…
Computational reproducibility is fundamental to trustworthy science, yet remains difficult to achieve in practice across various research workflows, including Jupyter notebooks published alongside scholarly articles. Environment drift,…
Global science is often portrayed as a unified system of shared knowledge and open exchange. Yet this vision contrasts with emerging evidence that scientific recognition is uneven and increasingly fragmented along regional and cultural…
The reproduction and replication of research results has become a major issue for a number of scientific disciplines. In computer science and related computational disciplines such as systems biology, the challenges closely revolve around…
Despite the vast literature on the diffusion of innovations that impacts a broad range of disciplines, our understanding of the abandonment of innovations remains limited yet is essential for a deeper understanding of the innovation…