Related papers: Interlaboratory consensus building challenge
Measurement uncertainty plays a critical role in the process of experimental physics. It is useful to be able to assess student proficiency around the topic to iteratively improve instruction and student learning. For the topic of…
Reliability is an essential measure of how closely observed scores represent latent scores (reflecting constructs), assuming some latent variable measurement model. We present a general theoretical framework of reliability, placing emphasis…
Large Language Models (LLMs) are increasingly embedded in academic writing practices. Although numerous studies have explored how researchers employ these tools for scientific writing, their concrete implementation, limitations, and design…
Technical standards are a longstanding method of communicating best practice recommendations based on expert consensus. Cybersecurity standards are particularly important for informing policies that protect critical systems and sensitive…
In this paper, the consensus problems of the continuous-time integrator systems under noisy measurements are considered. The measurement noises, which appear when agents measure their neighbors' states, are modeled to be multiplicative. By…
A confidence measure is able to estimate the reliability of an hypothesis provided by a machine translation system. The problem of confidence measure can be seen as a process of testing : we want to decide whether the most probable sequence…
We propose here selected actual features of measurement problems based on our concerns in our respective fields of research. Their technical similarity in apparently disconnected fields motivate this common communication. Problems of…
Consistency, defined as the requirement that a series of measurements of the same project carried out by different raters using the same method should produce similar results, is one of the most important aspects to be taken into account in…
By employing various empirical estimators for the Mutual Information (MI) measure, we calculate and compare the estimates and their confidence intervals for both normal and non-normal bivariate data samples. We find that certain nonlinear…
The task of natural language inference (NLI) asks whether a given premise (expressed in NL) entails a given NL hypothesis. NLI benchmarks contain human ratings of entailment, but the meaning relationships driving these ratings are not…
Judging the significance and reproducibility of quantitative research requires a good understanding of relevant uncertainties, but it is often unclear how well these have been evaluated and what they imply. Reported scientific uncertainties…
Robustness checks are routine in empirical work, but there is no standard statistical procedure to formally measure what one can learn from them. I propose a "robustness radius" measure to quantify the amount by which the robustness checks…
The basic idea of importance sampling is to use independent samples from a proposal measure in order to approximate expectations with respect to a target measure. It is key to understand how many samples are required in order to guarantee…
In addition to science citation indicators of journals like impact and immediacy, social network analysis provides a set of centrality measures like degree, betweenness, and closeness centrality. These measures are first analyzed for the…
The consensus problem -- achieving agreement among a network of agents -- is a central theme in both theory and applications. Recently, this problem has been extended from Euclidean spaces to the space of probability measures, where the…
In learning-assisted theorem proving, one of the most critical challenges is to generalize to theorems unlike those seen at training time. In this paper, we introduce INT, an INequality Theorem proving benchmark, specifically designed to…
Human annotation remains the foundation of reliable and interpretable data in Natural Language Processing (NLP). As annotation and evaluation tasks continue to expand, from categorical labelling to segmentation, subjective judgment, and…
This special volume of Statistical Sciences presents some innovative, if not provocative, ideas in the area of reliability, or perhaps more appropriately named, integrated system assessment. In this age of exponential growth in science,…
Metrics for measuring the comparability of corpora or texts need to be developed and evaluated systematically. Applications based on a corpus, such as training Statistical MT systems in specialised narrow domains, require finding a…
Rankings, representing preferences over a set of candidates, are widely used in many information systems, e.g., group decision making and information retrieval. It is of great importance to evaluate the consensus of the obtained rankings…