Related papers: Assessment and support of emerging research groups
There are various mathematical models proposed in the recent literature for estimating the h-index through bibliometric measures, such as number of articles (P) and citations received (C). These models have been previously empirically…
A bibliometric approach is explored to tracking international scientific migration, based on an analysis of the affiliation countries of authors publishing in peer reviewed journals indexed in Scopus. The paper introduces a model that…
A semi-supervised model of peer review is introduced that is intended to overcome the bias and incompleteness of traditional peer review. Traditional approaches are reliant on human biases, while consensus decision-making is constrained by…
The scientific system that we use today was devised centuries ago and is inadequate for our current ICT-based society: the peer review system encourages conservatism, journal publications are monolithic and slow, data is often not available…
Competitive allocation of research funding is a major mechanism within the science system. It is fundamentally based on the idea of peer review. Peer review is central in project selection as peers are considered to be in a unique position…
The last few years have seen the proliferation of measures that quantify the scientific output of researches. Yet, these measures focus on productivity, thus fostering the "publish or perish" paradigm. This article proposes a measure that…
Recent advancements in large language models have sparked interest in utilizing them to aid the peer review process of scientific publication amid the peer review crisis. However, having AI models generate full reviews in the same way as…
Comprehensive and accurate evaluation of general-purpose AI systems such as large language models allows for effective mitigation of their risks and deepened understanding of their capabilities. Current evaluation methodology, mostly based…
Classifying journals or publications into research areas is an essential element of many bibliometric analyses. Classification usually takes place at the level of journals, where the Web of Science subject categories are the most popular…
Large language models are proliferating, and so are the benchmarks that serve as their common yardsticks. We ask how the agglomeration patterns of these two layers compare: do they evolve in tandem or diverge? Drawing on two curated proxies…
This paper appraises the concordance between bibliometrics and peer review, by drawing evidence from the data of two experiments realized by the Italian governmental agency for research evaluation. The experiments were performed for…
LLM-based reasoning models have enabled the development of agentic systems that act as co-scientists, assisting in multi-step scientific analysis. However, evaluating these systems is challenging, as it requires realistic, end-to-end…
How much large language models (LLMs) can aid scientific discovery, notably in assisting academic peer review, is in heated debate. Between a literature digest and a human-comparable research assistant lies their practical application…
The analysis of research collaboration by field is traditionally conducted beginning with the classification of the publications from the context of interest. In this work we propose an alternative approach based on the classification of…
This paper empirically examines the practical validity of the official evaluation criteria underpinning the Research Productivity (PQ) Grant framework, as governed by the Brazilian National Council for Scientific and Technological…
The rise of agentic AI systems, where agents collaborate to perform diverse tasks, poses new challenges with observing, analyzing and optimizing their behavior. Traditional evaluation and benchmarking approaches struggle to handle the…
The rapid expansion of AI research has intensified the Reviewer Gap, threatening the peer-review sustainability and perpetuating a cycle of low-quality evaluations. This position paper critiques existing LLM approaches that automatically…
Over the past decades, the competition for academic resources has gradually intensified, and worsened with the current financial crisis. To optimize the resource allocation, individualized assessment of research results is being actively…
The ability to activate and manage effective collaborations is becoming an increasingly important criteria in policies on academic career advancement. The rise of such policies leads to development of indicators that permit measurement of…
This manuscript is a preprint version of Part 1 (General Introduction and Synopsis) of the book Applied Evaluative Informetrics, to be published by Springer in the summer of 2017. This book presents an introduction to the field of applied…