Related papers: The Evaluation Gap in Astronomy -- Explained throu…
Nine interviews were conducted with astronomers from Leiden University, and a document analysis was performed on relevant institutional (self-) evaluation documents, annual reports, and CVs of the interviewees. The aim was to perform a…
Recent research in the field of reflexive metrics have studied the emergence and consequences of evaluation gaps in science. The concept of evaluation gaps captures potential discrepancies between what researchers value about their…
Reflexive metrics is a branch of science studies which explores how the demand for accountability and performance measurement in science has shaped the research culture in recent decades. Hypercompetition and publication pressure are part…
The extensive focus on performance indicators in research evaluation has been facing critique in science studies. Stemming from a neoliberalist paradigm, metrics allegedly objectify and create certainty about researchers' performance. This…
Artificial intelligence is rapidly transforming astronomical research, yet the scientific community has largely treated this transformation as an engineering challenge rather than an epistemological one. This perspective article argues that…
About one fifth of Ph.D's across all STEM disciplines secure a tenure track position in academia. It is also the case that science and engineering have evolved significantly and so has the nature of the labor market and the increasingly…
In recent years, evaluation of the quality of academic research has become an increasingly important and influential business. It determines, often to a large extent, the amount of research funding flowing into universities and similar…
Observational astronomy is plagued with selection effects that must be taken into account when interpreting data from astronomical surveys. Because of the physical limitations of observing time and instrument sensitivity, datasets are…
The phenomenon of rankings is intimately related with the government interest in fiscalizing the research outputs of universities. New forms of managerialism have been introduced into the higher education system, leading to an increasing…
The astronomy community usually turns to the Astrophysics Data System for bibliometrics. When the context is cross-disciplinary, commercial products like Web of Science and Scopus are used along with related analytics tools instead. The…
Every AI benchmark operationalizes theoretical assumptions about the capability it claims to assess. When assumptions function as unexamined commitments, benchmarks stabilize the dominant paradigm by narrowing what counts as progress. Over…
A persistent challenge in astronomical machine learning is a systematic bias where predictions compress the dynamic range of true values-high values are consistently predicted too low while low values are predicted too high. Understanding…
National research evaluation exercises provide a comparative measure of research performance of the nation's institutions, and as such represent a tool for stimulating research productivity, particularly if the results are used to inform…
A common flaw of astronomers is to believe that they know the truth even when data is scarce. I provide ten examples where this approach led to wrong strategic decisions in research plans, causing unnecessary delays in finding the truth. In…
The recently published "certainty-scope" conjecture offers a compelling insight into the inherent trade-off present within artificial intelligence (AI) systems. As general research, this investigation remains vital as a philosophical…
A broad current application of algorithms is in formal and quantitative measures of murky concepts -- like merit -- to make decisions. When people strategically respond to these sorts of evaluations in order to gain favorable decision…
Forming a reliable judgement of a machine learning (ML) model's appropriateness for an application ecosystem is critical for its responsible use, and requires considering a broad range of factors including harms, benefits, and…
We use four years of introductory astronomy scores to analyze the ability of the current population to perform college level work and measure the amount of grade inflation across various majors. Using an objective grading scale, one that is…
Open Science has been a rising theme in the landscape of science policy in recent years. The goal is to make research that emerges from publicly funded science to become findable, accessible, interoperable and reusable (FAIR) for use by…
Anomalies drive scientific discovery -- they are associated with the cutting edge of the research frontier, and thus typically exploit data in the low signal-to-noise regime. In astronomy, the prevalence of systematics --- both "known…