Related papers: Statistical Significance Revisited
There is a well-known problem in Null Hypothesis Significance Testing: many statistically significant results fail to replicate in subsequent experiments. We show that this problem arises because standard `point-form null' significance…
Statistical significance testing of differences in values of metrics like recall, precision and balanced F-score is a necessary part of empirical natural language processing. Unfortunately, we find in a set of experiments that many commonly…
Importance sampling is a popular technique in Bayesian inference: by reweighting samples drawn from a proposal distribution we are able to obtain samples and moment estimates from a Bayesian posterior over latent variables. Recent work,…
Many Bayesian inference problems involve high dimensional models for which only a subset of the model variables are of actual interest. All other variables are just nuisance parameters that one would ideally like to integrate out…
In many fields of research null hypothesis significance tests and p values are the accepted way of assessing the degree of certainty with which research results can be extrapolated beyond the sample studied. However, there are very serious…
Statistical significance measures the reliability of a result obtained from a random experiment. We investigate the number of repetitions needed for a statistical result to have a certain significance. In the first step, we consider…
This document presents the statistical methods used to process low-level measurements in the presence of noise. These methods can be classical or Bayesian. The question is placed in the general framework of the problem of nuisance…
Bayesian hypothesis testing via Bayes factors offers a principled alternative to classical p-value methods in meta-analysis, particularly suited to its cumulative and sequential nature. Unlike commonly reported p-values for standard null…
Fisher's exact test is often a preferred method to estimate the significance of statistical dependence. However, in large data sets the test is usually too worksome to be applied, especially in an exhaustive search (data mining). The…
Fisher randomization tests for Neyman's null hypothesis of no average treatment effects are considered in a finite population setting associated with completely randomized experiments with more than two treatments. The consequences of using…
Computer experiments are becoming increasingly important in scientific investigations. In the presence of uncertainty, analysts employ probabilistic sensitivity methods to identify the key-drivers of change in the quantities of interest.…
Bayesian hypothesis testing is re-examined from the perspective of an a priori assessment of the test statistic distribution under the alternative. By assessing the distribution of an observable test statistic, rather than prior parameter…
Bayesian statistics is based on the subjective definition of probability as {\it ``degree of belief''} and on Bayes' theorem, the basic tool for assigning probabilities to hypotheses combining {\it a priori} judgements and experimental…
Null hypothesis statistical significance tests (NHST) are widely used in quantitative research in the empirical sciences including scientometrics. Nevertheless, since their introduction nearly a century ago significance tests have been…
In 1957, Lindley published "A statistical paradox" in Biometrika, revealing a fundamental conflict between frequentist and Bayesian inference as sample size approaches infinity. We present a new paradox of a different kind: a conflict…
Every scientific endeavour consists of (at least) two components: A hypothesis on the one hand and data on the other. There is always a more or less abstract level - some theory, a set of concepts, certain relations of ideas - and a…
The replication crisis has prompted many to call for statistical reform within the psychological sciences. Here we examine issues within Frequentist statistics that may have led to the replication crisis, and we examine the…
Statistical significance tests can provide evidence that the observed difference in performance between two methods is not due to chance. In Information Retrieval, some studies have examined the validity and suitability of such tests for…
Statistics has made tremendous advances since the times of Fisher, Neyman, Jeffreys, and others, but the fundamental and practically relevant questions about probability and inference that puzzled our founding fathers remain unanswered. To…
Outer measures can be used for statistical inference in place of probability measures to bring flexibility in terms of model specification. The corresponding statistical procedures such as Bayesian inference, estimators or hypothesis…