Related papers: Permutation-Based Tests of Perfect Ranking
Multiple-choice (MC) tests are an efficient method to assess English learners. It is useful for test creators to rank candidate MC questions by difficulty during exam curation. Typically, the difficulty is determined by having human test…
The problem of making practical, useful goodness of fit tests in the Bayesian paradigm is largely open. We introduce a class of special cases (testing for uniformity: have the cards been shuffled enough; does my random generator work) and a…
A new class of general exponential ranking models is introduced which we label angle-based models for ranking data. A consensus score vector is assumed, which assigns scores to a set of items, where the scores reflect a consensus view of…
We consider the problem of detecting the presence of a submatrix with larger-than-usual values in a large data matrix. This problem was considered in (Butucea and Ingster, 2013) under a one-parameter exponential family, and one of the test…
Invariance-based randomization tests -- such as permutation tests, rotation tests, or sign changes -- are an important and widely used class of statistical methods. They allow drawing inferences under weak assumptions on the data…
We introduce a method based on Conformal Prediction (CP) to quantify the uncertainty of full ranking algorithms. We focus on a specific scenario where $n+m$ items are to be ranked by some ``black box'' algorithm. It is assumed that the…
In randomized experiments with noncompliance, tests may focus on compliers rather than on the overall sample. Rubin (1998) put forth such a method, and argued that testing for the complier average causal effect and averaging permutation…
Large language models (LLMs) are now widely used to evaluate the quality of text, a field commonly referred to as LLM-as-a-judge. While prior works mainly focus on point-wise and pair-wise evaluation paradigms. Rubric-based evaluation,…
Ranking is a core task in recommender systems, which aims at providing an ordered list of items to users. Typically, a ranking function is learned from the labeled dataset to optimize the global performance, which produces a ranking score…
Smart word substitution aims to enhance sentence quality by improving word choices; however current benchmarks rely on human-labeled data. Since word choices are inherently subjective, ground-truth word substitutions generated by a small…
A consecutive pattern in a permutation $\pi$ is another permutation $\sigma$ determined by the relative order of a subsequence of contiguous entries of $\pi$. Traditional notions such as descents, runs and peaks can be viewed as particular…
We want to select the best systems out of a given set of systems (or rank them) with respect to their expected performance. The systems allow random observations only and we assume that the joint observation of the systems has a…
The permutation test is an often used test procedure in brain imaging. Unfortunately, generating every possible permutation for large-scale brain image datasets such as HCP and ADNI with hundreds images is not practical. Many previous…
Comparing the top $k$ elements between two or more ranked results is a common task in many contexts and settings. A few measures have been proposed to compare top $k$ lists with attractive mathematical properties, but they face a number of…
We compare different permutation tests and some parametric counterparts that are applicable to unbalanced designs in two by two designs. First the different approaches are shortly summarized. Then we investigate the behavior of the tests in…
We propose a class of locally and asymptotically optimal tests, based on multivariate ranks and signs for the homogeneity of scatter matrices in $m$ elliptical populations. Contrary to the existing parametric procedures, these tests remain…
Error bounds based on worst likely assignments use permutation tests to validate classifiers. Worst likely assignments can produce effective bounds even for data sets with 100 or fewer training examples. This paper introduces a statistic…
The paper deals with the problem of finding the best alternatives on the basis of pairwise comparisons when these comparisons need not be transitive. In this setting, we study a reinforcement urn model. We prove convergence to the optimal…
We describe a seriation algorithm for ranking a set of items given pairwise comparisons between these items. Intuitively, the algorithm assigns similar rankings to items that compare similarly with all others. It does so by constructing a…
The task of item recommendation requires ranking a large catalogue of items given a context. Item recommendation algorithms are evaluated using ranking metrics that depend on the positions of relevant items. To speed up the computation of…