Related papers: Forecast dominance testing via sign randomization
Recent studies have analyzed whether one forecast method dominates another under a class of consistent scoring functions. While the existing literature focuses on empirical tests of forecast dominance, little is known about the theoretical…
How can we monitor, in real time, whether one uncertain prospect has any upside over another? To answer this question, we develop a novel family of sequential, anytime-valid tests for stochastic dominance (SD; also known as stochastic…
This paper contains two finite-sample results concerning the sign test. First, we show that the sign-test is unbiased with independent, non-identically distributed data for both one-sided and two-sided hypotheses. The proof for the…
We review approaches to statistical inference based on randomization. Permutation tests are treated as an important special case. Under a certain group invariance property, referred to as the ``randomization hypothesis,'' randomization…
Stochastic dominance is a crucial tool for the analysis of choice under risk. It is typically analyzed as a property of two gambles that are taken in isolation. We study how additional independent sources of risk (e.g. uninsurable labor…
Scoring rules measure the deviation between a probabilistic forecast and reality. Strictly proper scoring rules have the property that for any forecast, the mathematical expectation of the score of a forecast p by the lights of p is…
Non-deterministic measurements are common in real-world scenarios: the performance of a stochastic optimization algorithm or the total reward of a reinforcement learning agent in a chaotic environment are just two examples in which…
We study how to perform tests on samples of pairs of observations and predictions in order to assess whether or not the predictions are prudent. Prudence requires that that the mean of the difference of the observation-prediction pairs can…
This paper proposes a new statistical test to assess the dominance of survival functions in the presence of right-censored data. Traditional methods, such as the log-rank test, are inadequate for determining whether one survival function…
Convex combinations of i.i.d. random variables without a finite mean can behave in a strikingly different way from the finite-mean case: as the weight vector becomes more balanced, the resulting combination may become stochastically larger,…
We provide conditions for the stochastic dominance comparisons of a risk $X$ and an associated risk $X+Z$, where $Z$ represents the uncertainty due to the environment and where $X$ and $Z$ can be dependent. The comparisons depend on both…
This paper deals with testing the equality of $k$ ($k\ge 2$) distribution functions against possible stochastic ordering among them. Two classes of rank tests are proposed for this testing problem. The statistics of the tests under study…
Robust classification algorithms have been developed in recent years with great success. We take advantage of this development and recast the classical two-sample test problem in the framework of classification. Based on the estimates of…
Instead of testing for unanimous agreement, I propose learning how broad of a consensus favors one distribution over another (of earnings, productivity, asset returns, test scores, etc.). Specifically, given a sample from each of two…
In observational studies of discrimination, the most common statistical approaches consider either the rate at which decisions are made (benchmark tests) or the success rate of those decisions (outcome tests). Both tests, however, have…
A number of applications require two-sample testing on ranked preference data. For instance, in crowdsourcing, there is a long-standing question of whether pairwise comparison data provided by people is distributed similar to…
A long noted difficulty when assessing the reliability (or calibration) of forecasting systems is that reliability, in general, is a hypothesis not about a finite dimensional parameter but about an entire functional relationship. A…
Predictions about people, such as their expected educational achievement or their credit risk, can be performative and shape the outcome that they aim to predict. Understanding the causal effect of these predictions on the eventual outcomes…
A benefit of randomized experiments is that covariate distributions of treatment and control groups are balanced on average, resulting in simple unbiased estimators for treatment effects. However, it is possible that a particular…
We compare probabilistic predictions of extreme temperature anomalies issued by two different forecast schemes. One is a dynamical physical weather model, the other a simple data model. We recall the concept of skill scores in order to…