Related papers: Testing Rankings with Cross-Validation

We improve three tests of perfect ranking in ranked set sampling proposed by Li and Balakrishnan (2008) using a permutation approach. This simple way of extending all three concepts to comparisons across different cycles increases the…

Randomized artificial neural networks such as extreme learning machines provide an attractive and efficient method for supervised learning under limited computing ressources and green machine learning. This especially applies when equipping…

Cross-Validation (CV), and out-of-sample performance-estimation protocols in general, are often employed both for (a) selecting the optimal combination of algorithms and values of hyper-parameters (called a configuration) for producing the…

With the increasing size of today's data sets, finding the right parameter configuration in model selection via cross-validation can be an extremely time-consuming task. In this paper we propose an improved cross-validation procedure which…

Randomized clinical trials (RCTs) are widely considered the gold standard for evaluating the effectiveness of new treatments or interventions in drug development. Still, they may not be feasible in certain cases, such as with rare diseases…

The stochastic block model and its variants have been a popular tool in analyzing large network data with community structures. In this paper we develop an efficient network cross-validation (NCV) approach to determine the number of…

This paper addresses feature subset selection for Support Vector Machines (SVMs) based on the cross-validation criterion. Unlike statistical criteria such as the Akaike information criterion (AIC) and the Bayesian information criterion…

Analyses of high-throughput genomic data often lead to ranked lists of genomic loci. How to characterize concordant signals between two rank lists is a common problem with many applications. One example is measuring the reproducibility…

Survival analysis often relies on Cox models, assuming both linearity and proportional hazards (PH). This study evaluates machine and deep learning methods that relax these constraints, comparing their performance with penalized Cox models…

In this paper we develop a novel nonparametric framework to test the independence of two random variables $\mathbf{X}$ and $\mathbf{Y}$ with unknown respective marginals $H(dx)$ and $G(dy)$ and joint distribution $F(dx dy)$, based on {\it…

We leverage recent advances in heavy-tail approximations for global hypothesis testing with dependent studies to construct approximate confidence regions without modeling or estimating their dependence structures. A non-rejection region is…

We consider Wald's sequential probability ratio test for deciding whether a sequence of independent and identically distributed observations comes from a specified phase-type distribution or from an exponentially tilted alternative…

Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter…

Learning to Rank (LTR) methods generally assume that each document in a top-K ranking is presented in an equal format. However, previous work has shown that users' perceptions of relevance can be changed by varying presentations, i.e.,…

In a recent paper entitled "Inconsistencies of Recently Proposed Citation Impact Indicators and how to Avoid Them," Schreiber (2012, at arXiv:1202.3861) proposed (i) a method to assess tied ranks consistently and (ii) fractional attribution…

In classification problems, sampling bias between training data and testing data is critical to the ranking performance of classification scores. Such bias can be both unintentionally introduced by data collection and intentionally…

Network-valued data are encountered in a wide range of applications and pose challenges in learning due to their complex structure and absence of vertex correspondence. Typical examples of such problems include classification or grouping of…

A popular approach for testing if two univariate random variables are statistically independent consists of partitioning the sample space into bins, and evaluating a test statistic on the binned data. The partition size matters, and the…

Multiple-choice exams are frequently used as an efficient and objective method to assess learning but they are more vulnerable to answer-copying than tests based on open questions. Several statistical tests (known as indices in the…

Ensemble disagreement is widely used as a proxy for epistemic uncertainty in medical image segmentation. In practice, many studies form ensembles via K-fold cross-validation (CV), yet refer to them as ``deep ensembles'' (DE). Because CV…