Related papers: From Risk Measures to Research Measures
Quantifying success in science plays a key role in guiding funding allocations, recruitment decisions, and rewards. Recently, a significant amount of progresses have been made towards quantifying success in science. This lack of detailed…
We present a novel approach to visually locate bodies of research within the sciences, both at each moment of time and dynamically. This article describes how this approach fits with other efforts to locally and globally map scientific…
Mapping Science across countries is a challenging task in the field of Scientometrics. A number of efforts trying to cope with this task has been discussed in the state of the art, addressing this challenge by processing collections of…
Scientific problem solving poses unique challenges for LLMs, requiring both deep domain knowledge and the ability to apply such knowledge through complex reasoning. While automated scientific reasoners hold great promise for assisting human…
The use of metrics underpins the quantification, communication and, ultimately, the functioning of a wide range of disciplines as diverse as labour recruitment, institutional management, economics and science. For application of metrics,…
Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-based systems, but developers nonetheless claim that their…
Recent work on algorithmic fairness has largely focused on the fairness of discrete decisions, or classifications. While such decisions are often based on risk score models, the fairness of the risk models themselves has received…
The impact of individual scientists is commonly quantified using citation-based measures. The most common such measure is the h-index. A scientist's h-index affects hiring, promotion, and funding decisions, and thus shapes the progress of…
Systemic risk measures were introduced to capture the global risk and the corresponding contagion effects that is generated by an interconnected system of financial institutions. To this purpose, two approaches were suggested. In the first…
We discuss a recently proposed family of statistical network models - relational hyperevent models (RHEM) - for analyzing team selection and team performance in scientific coauthor networks. The underlying rationale for using RHEM in…
The summarization capabilities of pretrained and large language models (LLMs) have been widely validated in general areas, but their use in scientific corpus, which involves complex sentences and specialized knowledge, has been less…
Many theories of scientific and technological progress imagine science as an iterative, developmental process periodically interrupted by innovations which disrupt and restructure the status quo. Due to the immense societal value created by…
In this paper, we introduce a new class of set-valued risk measures, named set-valued star-shaped risk measures. Motivated by the results of scalar monetary and star-shaped risk measures, this paper investigates the representation theorems…
In scientific research and its application, scientific literature analysis is crucial as it allows researchers to build on the work of others. However, the fast growth of scientific knowledge has led to a massive increase in scholarly…
The proliferation of LLM bias probes introduces three significant challenges: (1) we lack principled criteria for choosing appropriate probes, (2) we lack a system for reconciling conflicting results across probes, and (3) we lack formal…
The number of Language Models (LMs) dedicated to processing scientific text is on the rise. Keeping pace with the rapid growth of scientific LMs (SciLMs) has become a daunting task for researchers. To date, no comprehensive surveys on…
Using ideas and tools of complexity science we design a holistic measure of \textit{Scientific Fitness}, encompassing the scientific knowledge, capabilities and competitiveness of a research system. We characterize the temporal dynamics of…
Scientometric data is used to investigate empirically the emergence of search regimes in Biotechnology, Genomics, and Nanotechnology. Complex regimes can emerge when three independent sources of variance interact. In our model, researchers…
Context: Systematic literature reviews (SLRs) have become standard practise as part of software engineering research, although their quality varies. To build on the reviews, both for future research and industry practice, they need to be of…
There is a worldwide trend towards application of bibliometric research evaluation, in support of the needs of policy makers and research administrators. However the assumptions and limitations of bibliometric measurements suggest a…