Related papers: On Statistical Non-Significance
Textbooks on statistics emphasize care and precision, via concepts such as reliability and validity in measurement, random sampling and treatment assignment in data collection, and causal identification and bias in estimation. But how do…
Despite its importance to experimental design, statistical power (the probability that, given a real effect, an experiment will reject the null hypothesis) has largely been ignored by the NLP community. Underpowered experiments make it more…
Authors often struggle to interpret peer review feedback, deriving false hope from polite comments or feeling confused by specific low scores. To investigate this, we construct a dataset of over 30,000 ICLR 2021-2025 submissions and compare…
Verifying that a statistically significant result is scientifically meaningful is not only good scientific practice, it is a natural way to control the Type I error rate. Here we introduce a novel extension of the p-value - a…
To properly estimate signal significance while accounting for both statistical and systematic uncertainties, we conducted a study to analyze the impact of typical systematic uncertainties, such as background shape, signal shape, and the…
Publications proposing novel machine learning methods are often primarily rated by exhibited predictive performance on selected problems. In this position paper we argue that predictive performance alone is not a good indicator for the…
In mathematics information is a number that measures uncertainty (entropy) based on a probabilistic distribution, often of an obscure origin. In real life language information is a datum, a statement, more precisely, a formula. But such a…
The Full Bayesian Significance Test (FBST) for precise hypotheses was presented by Pereira and Stern (1999) as a Bayesian alternative instead of the traditional significance test based on p-value. The FBST uses the evidence in favor of the…
Information integration plays a pivotal role in biomedical studies by facilitating the combination and analysis of independent datasets from multiple studies, thereby uncovering valuable insights that might otherwise remain obscured due to…
A sequence of recent papers has considered the role of measurement scales in information retrieval (IR) experimentation, and presented the argument that (only) uniform-step interval scales should be used, and hence that well-known metrics…
It is possible to obtain a large Bayes Factor (BF) favoring the null hypothesis when both the null and alternative hypotheses have low likelihoods, and there are other hypotheses being ignored that are much more strongly supported by the…
In many research contexts, especially in the biomedical field, after studying and developing a classification system a natural question arises: "Is this accuracy enough high?", or better, "Can we say, with a statistically significant…
We consider the power to reject false values of the parameter in Frequentist methods for the calculation of confidence intervals. We connect the power with the physical significance (reliability) of confidence intervals for a parameter…
The application of machine learning based decision making systems in safety critical areas requires reliable high certainty predictions. Reject options are a common way of ensuring a sufficiently high certainty of predictions made by the…
A dimension reduction-based adaptive-to-model test is proposed for significance of a subset of covariates in the context of a nonparametric regression model. Unlike existing local smoothing significance tests, the new test behaves like a…
Importance sampling is often used in machine learning when training and testing data come from different distributions. In this paper we propose a new variant of importance sampling that can reduce the variance of importance sampling-based…
Importance weighting is a fundamental procedure in statistics and machine learning that weights the objective function or probability distribution based on the importance of the instance in some sense. The simplicity and usefulness of the…
Statistical hypothesis tests typically use prespecified sample sizes, yet data often arrive sequentially. Interim analyses invalidate classical error guarantees, while existing sequential methods require rigid testing preschedules or incur…
Determining the relevant spatial covariates is one of the most important problems in the analysis of point patterns. Parametric methods may lead to incorrect conclusions, especially when the model of interactions between points is wrong.…
We are concerned with the problem of introducing credibility type information into reasoning systems. The concept of credibility allows us to discount information provided by agents. An important characteristic of this kind of procedure is…