Related papers: Stop Using the Wilcoxon Test: Myth, Misconception …
The rank correlation coefficients and the ranked-based statistical tests (as a subset of non-parametric techniques) might be misleading when they are applied to subjectively collected opinion scores. Those techniques assume that the data is…
Ranked lists are frequently used by information retrieval (IR) systems to present results believed to be relevant to the users information need. Fairness is a relatively new but important aspect of these rankings to measure, joining a rich…
Confronted with the challenge of identifying the most suitable metric to validate the merits of newly proposed models, the decision-making process is anything but straightforward. Given that comparing rankings introduces its own set of…
Online experiments are widely used for improving online services. While doing online experiments, The student t-test is the most widely used hypothesis testing technique. In practice, however, the normality assumption on which the t-test…
It is quite common in modern research, for a researcher to test many hypotheses. The statistical (frequentist) hypothesis testing framework, does not scale with the number of hypotheses in the sense that naively performing many hypothesis…
A sequence of recent papers has considered the role of measurement scales in information retrieval (IR) experimentation, and presented the argument that (only) uniform-step interval scales should be used, and hence that well-known metrics…
This note revisits Steel's multiple comparison test which uses Wilcoxon statistics in pairwise comparisons of several treatment samples with a common control sample. It derives means, variances and covariances of the Wilcoxon statistics…
In scientific research, many hypotheses relate to the comparison of two independent groups. Usually, it is of interest to use a design (i.e., the allocation of sample sizes $m$ and $n$ for fixed $N = m + n$) that maximizes the power of the…
We propose new smoothed median and the Wilcoxon's rank sum test. As is pointed out by Maesono et al.(2016), some nonparametric discrete tests have a problem with their significance probability. Because of this problem, the selection of the…
In evaluation campaigns, participants often explore variations of popular, state-of-the-art baselines as a low-risk strategy to achieve competitive results. While effective, this can lead to local "hill climbing" rather than more radical…
Rank-based inference methods are applied in various disciplines, typically when procedures relying on standard normal theory are not justifiable, for example when data are not symmetrically distributed, contain outliers, or responses are…
Replicability and reproducibility of experimental results are primary concerns in all the areas of science and IR is not an exception. Besides the problem of moving the field towards more reproducible experimental practices and protocols,…
The Wilcoxon-Mann-Whitney test is a robust competitor of the t-test in the univariate setting. For finite dimensional multivariate data, several extensions of the Wilcoxon-Mann-Whitney test have been shown to have better performance than…
The evaluation of Information Retrieval (IR) systems typically uses query-document pairs with corresponding human-labelled relevance assessments (qrels). These qrels are used to determine if one system is better than another based on…
Several tasks in information retrieval (IR) rely on assumptions regarding the distribution of some property (such as term frequency) in the data being processed. This thesis argues that such distributional assumptions can lead to incorrect…
In this paper we propose new smoothed sign and Wilcoxon's signed rank tests, which are based on a kernel estimator of the underlying distribution function of data. We discuss approximations of $p$-values and asymptotic properties of these…
Security especially in the fields of IoT, industrial automation and critical infrastructure is paramount nowadays and a hot research topic. In order to ensure confidence in research results they need to be reproducible. In the past we…
This paper considers the problem of testing for latent structure in large symmetric data matrices. The goal here is to develop statistically principled methodology that is flexible in its applicability, computationally efficient, and…
Many testing problems are readily amenable to randomised tests such as those employing data splitting. However despite their usefulness in principle, randomised tests have obvious drawbacks. Firstly, two analyses of the same dataset may…
We propose a testing procedure based on the Wilcoxon two-sample test statistic in order to test for change-points in the mean of long-range dependent data. We show that the corresponding self-normalized test statistic converges in…