Related papers: Significance Tests in Climate Science
Since its introduction by Fisher, the method of hypothesis testing that relies on computing error probabilities has witnessed several developments. Perhaps the most significant development was the seminal contributions of Neyman and Pearson…
In cell biology, statistical analysis means testing the hypothesis that there was no effect. This weak form of hypothesis testing neglects effect size, is universally misinterpreted, and is disastrously prone to error when combined with…
Statistical methods are indispensable to scientific inference. However, there exists a longstanding tension across a wide range of scientific disciplines about the role that ``context'' should play in the application of statistical methods…
Most scientific disciplines use significance testing to draw conclusions about experimental or observational data. This classical approach provides a theoretical guarantee for controlling the number of false positives across a set of…
Probability models are only useful at explaining the uncertainty of what we do not know, and should never be used to say what we already know. Probability and statistical models are useless at discerning cause. Classical statistical…
We examine the role of trustworthiness and trust in statistical inference, arguing that it is the extent of trustworthiness in inferential statistical tools which enables trust in the conclusions. Certain tools, such as the p-value and…
The "significance filter" refers to focusing exclusively on statistically significant results. Since frequentist properties such as unbiasedness and coverage are valid only before the data have been observed, there are no guarantees if we…
Statistical significance testing is used in natural language processing (NLP) to determine whether the results of a study or experiment are likely to be due to chance or if they reflect a genuine relationship. A key step in significance…
In this manuscript, we discuss the substantial importance of Bayesian reasoning in Social Science research. Particularly, we focus on foundational elements to fit models under the Bayesian paradigm. We aim to offer a frame of reference for…
Often information relevant to a decision is summarized in an index number. This paper explores conditions under which conclusions using index numbers are relevant to the decision that needs to be made. Specifically it explores the idea that…
Condensing the work of any academic scientist into a one-dimensional measure of scientific quality is a difficult problem. Here, we employ Bayesian statistics to analyze several different measures of quality. Specifically, we determine each…
Bayes factors have become a popular tool in exoplanet spectroscopy for testing atmosphere models against one another. We show that the commonly used method for converting these values into significance "sigmas" is invalid. The formula is…
To properly estimate signal significance while accounting for both statistical and systematic uncertainties, we conducted a study to analyze the impact of typical systematic uncertainties, such as background shape, signal shape, and the…
A researcher is interested in what sample size is needed to get the required significance of the same test, assuming exactly the same situation that was in the study with the non-significant result. We propose a simple solution to the…
Statistical significance tests can provide evidence that the observed difference in performance between two methods is not due to chance. In Information Retrieval, some studies have examined the validity and suitability of such tests for…
Like it or not, attempts to evaluate and monitor the quality of academic research have become increasingly prevalent worldwide. Performance reviews range from at the level of individuals, through research groups and departments, to entire…
The established language for statistical testing --- significance levels, power, and p-values --- is overly complicated and deceptively conclusive. Even teachers of statistics and scientists who use statistics misinterpret the results of…
Recently, it was shown that most popular IR measures are not interval-scaled, implying that decades of experimental IR research used potentially improper methods, which may have produced questionable results. However, it was unclear if and…
A key goal of empirical research in software engineering is to assess practical significance, which answers whether the observed effects of some compared treatments show a relevant difference in practice in realistic scenarios. Even though…
In a Forum published in EOS Transactions AGU (2009) entitled "Lies, damned lies and statistics (in Geology)", Vermeesch (2009) claims that "statistical significant is not the same as geological significant", in other words, statistical…