Related papers: The Evaluation Gap in Astronomy -- Explained throu…
Selective classifiers improve model reliability by abstaining on inputs the model deems uncertain. However, few practical approaches achieve the gold-standard performance of a perfect-ordering oracle that accepts examples exactly in order…
This paper presents a praxeological analysis of artificial intelligence and algorithmic governance, challenging assumptions about the capacity of machine systems to sustain economic and epistemic order. Drawing on Misesian a priori…
Ranking and scoring are ubiquitous. We consider the setting in which an institution, called a ranker, evaluates a set of individuals based on demographic, behavioral or other characteristics. The final output is a ranking that represents…
The acclaimed merits of analytical solutions based on a fictitious time developed in the 1970's were partially overvalued due to a common misuse of classical analytical solutions based on the physical time that were taken as reference. With…
Research performance is often measured using bibliometric indicators, such as publication count, total citations, and $h$-index. These metrics influence career advancements, salary adjustments, administrative opportunities, funding…
[Context] Quality requirements are important for product success yet often handled poorly. The problems with scope decision lead to delayed handling and an unbalanced scope. [Objective] This study characterizes the scope decision process to…
One of the key skills of a researcher is noticing what's going on. Both in the experiment one's performing and in one's data: is there something interesting, reason to doubt one's data or suspect that one's theoretical description is…
In order to solve a task using reinforcement learning, it is necessary to first formalise the goal of that task as a reward function. However, for many real-world tasks, it is very difficult to manually specify a reward function that never…
Science is becoming increasingly more interdisciplinary, giving rise to more diversity in the areas of expertise within research labs and groups. This also have brought changes to the role researchers in scientific works. As a consequence,…
Governance theory has quietly relied on a rough cognitive comparability between governors and governed. The assumption is load-bearing, and this paper tries to show why by making it testable. The vehicle is a six-dimension evaluation…
The study of defeasible reasoning unites epistemologists with those working in AI, in part, because both are interested in epistemic rationality. While it is traditionally thought to govern the formation and (with)holding of beliefs,…
Philosophers have recently focused on critical, epistemological challenges that arise from the opacity of deep neural networks. One might conclude from this literature that doing good science with opaque models is exceptionally challenging,…
Problems for evaluation and impact of published scientific works and their authors are discussed. The role of citations in this process is pointed out. Different bibliometric indicators are reviewed in this connection and ways for…
We discuss the use of the Bayesian evidence ratio, or Bayes factor, for model selection in astronomy. We treat the evidence ratio as a statistic and investigate its distribution over an ensemble of experiments, considering both simple…
Existing research primarily evaluates the values of LLMs by examining their stated inclinations towards specific values. However, the "Value-Action Gap," a phenomenon rooted in environmental and social psychology, reveals discrepancies…
This paper proposes a suite of rationality measures and associated theory for reinforcement learning agents, a property increasingly critical yet rarely explored. We define an action in deployment to be perfectly rational if it maximises…
Traditionally, the way one evaluates the performance of an Artificial Intelligence (AI) system is via a comparison to human performance in specific tasks, treating humans as a reference for high-level cognition. However, these comparisons…
Cosmology contributes a good deal to the investigation of variation of fundamental physical constants. High resolution data is available and allows for detailed analysis over cosmological distances and a multitude of methods were developed.…
Understanding why a spacecraft maneuvers -- rather than simply that it did -- is an increasingly important problem for space domain awareness as Earth orbits grow crowded and contested. Current analysis pipelines are built for detection:…
Traditional closed peer review systems, which have played a central role in scientific publishing, are often slow, costly, non-transparent, stochastic, and possibly subject to biases - factors that can impede scientific progress and…