Related papers: P-values: misunderstood and misused
This is a report about the use and misuse of citation data in the assessment of scientific research. The idea that research assessment must be done using ``simple and objective'' methods is increasingly prevalent today. The ``simple and…
The F-measure or F-score is one of the most commonly used single number measures in Information Retrieval, Natural Language Processing and Machine Learning, but it is based on a mistake, and the flawed assumptions render it unsuitable for…
A new percentile-based rating scale P100 has recently been proposed to describe the citation impact in terms of the distribution of the unique citation values. Here I investigate P100 for 5 example datasets, two simple fictitious models and…
Note: Published now as a chapter in "Handbook of the History and Philosophy of Mathematical Practice" (Springer Nature, editor B. Sriraman, https://doi.org/10.1007/978-3-030-19071-2_105-1). The application of mathematical probability theory…
This essay looks at decision-making with interval-valued probability measures. Existing decision methods have either supplemented expected utility methods with additional criteria of optimality, or have attempted to supplement the…
Because there are similarities between the evaluation of alternative stories in criminal trials and the evaluation of scientific theories, scholars have looked to literature in epistemology and the philosophy of science for insights on the…
Confidence intervals are a popular way to visualize and analyze data distributions. Unlike p-values, they can convey information both about statistical significance as well as effect size. However, very little work exists on applying…
This paper studies the construction of p-values for nonparametric outlier detection, taking a multiple-testing perspective. The goal is to test whether new independent samples belong to the same distribution as a reference data set or are…
The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new…
Clinical research should conform to high standards of ethical and scientific integrity, given that human lives are at stake. However, economic incentives can generate conflicts of interest for investigators, who may be inclined to withhold…
Percentage coefficient (bp) has emerged in recent publications as an additional and alternative estimator of effect size for regression analysis. This paper retraces the theory behind the estimator. It's posited that an estimator must first…
We develop the distribution of the number of hypotheses found to be statistically significant using the rule from Benjamini and Hochberg (1995) for controlling the false discovery rate (FDR). This distribution has both a small sample form…
In the final analysis citation-based indicators are inferior to effective peer review and even peer review is flawed. It is impossible to accurately measure the value or impact of scientific research and a key task of scientometricians…
In experiments where one searches a large parameter space for an anomaly, one often finds many spurious noise-induced peaks in the likelihood. This is known as the look-elsewhere effect, and must be corrected for when performing statistical…
Graphical tests assess whether a function of interest departs from an envelope of functions generated under a simulated null distribution. This approach originated in spatial statistics, but has recently gained some popularity in functional…
Cluster analysis is a fundamental research issue in statistics and machine learning. In many modern clustering methods, we need to determine whether two subsets of samples come from the same cluster. Since these subsets are usually…
Scientific experiments study interventions that show evidence of an effect size that is meaningfully large, negligibly small, or inconclusively broad. Previously, we proposed contra-analysis as a decision-making process to help determine…
$P$-values that are derived from continuously distributed test statistics are typically uniformly distributed on $(0,1)$ under least favorable parameter configurations (LFCs) in the null hypothesis. Conservativeness of a $p$-value $P$…
A lot of Machine Learning (ML) and Deep Learning (DL) research is of an empirical nature. Nevertheless, statistical significance testing (SST) is still not widely used. This endangers true progress, as seeming improvements over a baseline…
Combining p-values from independent statistical tests is a popular approach to meta-analysis, particularly when the data underlying the tests are either no longer available or are difficult to combine. A diverse range of p-value combination…