Related papers: Measuring a moving target -- Innovation studies in…
While the methodological rigor of computing research has improved considerably in the past two decades, quantitative software engineering research is hampered by immature measures and inattention to theory. Measurement-the principled…
How much does a research study contribute to a scientific literature? We propose a learning metric to quantify how much a research community learns from a given study. To do so, we adopt a Bayesian perspective and assess changes in the…
The development of scientometric indicators and methods for evaluative purposes, requires a multitude of assumptions, conventions, limitations, and caveats. Given this, we cannot permit ambiguities in the key concepts forming the basis of…
Scientometrics is the study of the quantitative aspects of the process of science as a communication system. It is centrally, but not only, concerned with the analysis of citations in the academic literature. In recent years it has come to…
Scholarly usage data holds the potential to be used as a tool to study the dynamics of scholarship in real time, and to form the basis for the definition of novel metrics of scholarly impact. However, the formal groundwork to reliably and…
Measuring innovation often relies on context-specific proxies and on expert evaluation. Hence, empirical innovation research is often limited to settings where such data is available. We investigate how large language models (LLMs) can be…
Measuring performance & quantifying a performance change are core evaluation techniques in programming language and systems research. Of 122 recent scientific papers, as many as 65 included experimental evaluation that quantified a…
We will outline our ideas for teaching in the core mathematics disciplines. They are based on our own experience in teaching at a number of universities in the USA, as well as in Europe. While some of the core ideas stay and have stayed…
What science does, what science could do, and how to make science work? If we want to know the answers to these questions, we need to be able to uncover the mechanisms of science, going beyond metrics that are easily collectible and…
The paper provides an overview of the field of scientometrics, that is: the study of science, technology, and innovation from a quantitative perspective. We cover major historical milestones in the development of this specialism from the…
What kind of basic research ideas are more likely to get applied in practice? There is a long line of research investigating patterns of knowledge transfer, but it generally focuses on documents as the unit of analysis and follow their…
In many scientific and data-driven applications, machine learning models are increasingly used as measurement instruments, rather than merely as predictors of predefined labels. When the measurement function is learned from data, the…
One of the key skills of a researcher is noticing what's going on. Both in the experiment one's performing and in one's data: is there something interesting, reason to doubt one's data or suspect that one's theoretical description is…
New ideas are often thought to arise from recombining existing knowledge. Yet despite rapid publication growth - and expanding opportunities for recombination - scientific breakthroughs remain rare. This gap between productivity and…
Organized knowledge production can then be considered as the codification of communication. Communications leave traces that can be studied as indicators. Institutions can be considered as retention mechanisms functional for the…
The tension between qualitative theorizing and quantitative methods is pervasive in the social sciences, and poses a constant challenge to empirical research. But in science studies as an interdisciplinary specialty, there are additional…
Like it or not, attempts to evaluate and monitor the quality of academic research have become increasingly prevalent worldwide. Performance reviews range from at the level of individuals, through research groups and departments, to entire…
Novelty evaluation is vital for the promotion and management of innovation. With the advancement of information techniques and the open data movement, some progress has been made in novelty measurements. Tracking and reviewing novelty…
Progress in science and technology is punctuated by disruptive innovation and breakthroughs. Researchers have characterized these disruptions to explore the factors that spark such innovations and to assess their long-term trends. However,…
Mathematical modelling is at the core of metrology as it transforms raw measured data into useful measurement results. A model captures the relationship between the measurand and all relevant quantities on which the measurand depends, and…