Related papers: Assessment and support of emerging research groups
Using bibliometric data artificially generated through a model of citation dynamics calibrated on empirical data, we compare several indicators for the scientific impact of individual researchers. The use of such a controlled setup has the…
Bibliometrics is successful in measuring impact, because the target is clearly defined: the publishing scientist who is still active and working. Thus, citations are a target-oriented metric which measures impact on science. In contrast,…
The rapid adoption of AI agents across domains has made systematic evaluation crucial for ensuring their usefulness and successful production deployment. Evaluation of AI agents typically involves using a fixed set of benchmarks and…
Bibliometric indicators such as journal impact factors, h-indices, and total citation counts are algorithmic artifacts that can be used in research evaluation and management. These artifacts have no meaning by themselves, but receive their…
The basic indicators of a researcher's productivity and impact are still the number of publications and their citation counts. These metrics are clear, straightforward, and easy to obtain. When a ranking of scholars is needed, for instance…
We propose an academic publishing system where research papers are stored in a network of data centres owned by university libraries and research institutions, and are interfaced with the academic community through a website. In our system,…
In recent years, new indicators known as altmetrics have been introduced to measure the impact of scientific activity. These indicators are obtained through the mentions realised from different social media, existing several aggregators of…
As Large Language Models (LLMs) transition from static tools to autonomous agents, traditional evaluation benchmarks that measure performance on downstream tasks are becoming insufficient. These methods fail to capture the emergent social…
In this chapter, we focus on the specialists who have helped to improve the conditions for book assessments in research evaluation exercises, with empirically based data and insights supporting their greater integration. Our review…
The use of quantitative indicators of scientific productivity seems now quite widespread for assessing researchers and research institutions. There is a general perception, however, that these indicators are not necessarily representative…
It is becoming ever more common to use bibliometric indicators to evaluate the performance of research institutions, however there is often a failure to recognize the limits and drawbacks of such indicators. Since performance measurement is…
Modern information access ecosystems consist of mixtures of systems, such as retrieval systems and large language models, and increasingly rely on marketplaces to mediate access to models, tools, and data, making competition between systems…
The recognition of academic excellence is fundamental to the scientific and academic endeavor. However, the term "academic excellence" is often interpreted in different ways, typically, using popular scientometrics such as the H-index,…
Finding new academic Methods for research problems is the key task in a researcher's research career. It is usually very difficult for new researchers to find good Methods for their research problems since they lack of research experiences.…
In the context of cluster analysis and graph partitioning, many external evaluation measures have been proposed in the literature to compare two partitions of the same set. This makes the task of selecting the most appropriate measure for a…
The deluge of new papers has significantly blocked the development of academics, which is mainly caused by author-level and publication-level evaluation metrics that only focus on quantity. Those metrics have resulted in several severe…
Systematic evaluations of publicly funded research typically employ a combination of bibliometrics and peer review, but it is not known whether the bibliometric component introduces biases. This article compares three alternative mechanisms…
Development of bibliometric techniques has reached such a level as to suggest their integration or total substitution for classic peer review in the national research assessment exercises, as far as the hard sciences are concerned. In this…
After almost four decades of participating in competitive research funding -- as applicant, coordinator, evaluator, and panel member -- I have come to see a structural paradox: many participants recognize that the current system is…
This position paper argues that the under-representation of social science tasks in contemporary LLM benchmarks limits advances in both LLM evaluation and social scientific inquiry. Benchmarks -- standardized tools for assessing…