Related papers: Confidence intervals for class prevalences under p…
A prediction interval covers a future observation from a random process in repeated sampling, and is typically constructed by identifying a pivotal quantity that is also an ancillary statistic. Analogously, a tolerance interval covers a…
Real-world deployment of machine learning models is challenging because data evolves over time. While no model can work when data evolves in an arbitrary fashion, if there is some pattern to these changes, we might be able to design methods…
Constructing confidence intervals that are simultaneously valid across a class of estimates is central to tasks such as multiple mean estimation, generalization guarantees, and adaptive experimental design. We frame this as an ``error…
We consider interval estimation of the difference between two binomial proportions. Several methods of constructing such an interval are known. Unfortunately those confidence intervals have poor coverage probability: it is significantly…
The quantification problem consists of determining the prevalence of a given label in a target population. However, one often has access to the labels in a sample from the training population but not in the target population. A common…
The behaviors of various confidence/credible interval constructions are explored, particularly in the region of low statistics where methods diverge most. We highlight a number of challenges, such as the treatment of nuisance parameters,…
This paper generalizes the traditional statistical concept of prediction intervals for arbitrary probability density functions in high-dimensional feature spaces by introducing significance level distributions, which provides…
Estimating the prevalence of a category in a population using imperfect measurement devices (diagnostic tests, classifiers, or large language models) is fundamental to science, public health, and online trust and safety. Standard approaches…
We compare several confidence intervals after model selection in the setting recently studied by Berk et al. [Ann. Statist. 41 (2013) 802-837], where the goal is to cover not the true parameter but a certain nonstandard quantity of interest…
We study the frequentist properties of confidence intervals computed by the method known to statisticians as the Profile Likelihood. It is seen that the coverage of these intervals is surprisingly good over a wide range of possible…
In this article, we derive an explicit formula for computing confidence interval for the mean of a bounded random variable. Moreover, we have developed multistage point estimation methods for estimating the mean value with prescribed…
Turing's estimator allows one to estimate the probabilities of outcomes that either do not appear or only rarely appear in a given random sample. We perform a simulation study to understand the finite sample performance of several related…
We show that in the context of classification the property of source and target distributions to be related by covariate shift may be lost if the information content captured in the covariates is reduced, for instance by dropping components…
Several scientific fields including psychology are undergoing a replication crisis. There are many reasons for this problem, one of which is a misuse of p-values. There are several alternatives to p-values, and in this paper we describe a…
Prediction, where observed data is used to quantify uncertainty about a future observation, is a fundamental problem in statistics. Prediction sets with coverage probability guarantees are a common solution, but these do not provide…
In many statistical problems, several estimators are usually available for interval estimation of a parameter of interest, and hence, the selection of an appropriate estimator is important. The criterion for a good estimator is to have a…
When facing uncertainty, decision-makers want predictions they can trust. A machine learning provider can convey confidence to decision-makers by guaranteeing their predictions are distribution calibrated -- amongst the inputs that receive…
In computational social science, researchers often use a pre-trained, black box classifier to estimate the frequency of each class in unlabeled datasets. A variety of prevalence estimation techniques have been developed in the literature,…
Machine learning models are often evaluated using point estimates of performance metrics such as accuracy, F1 score, or mean squared error. Such summaries fail to capture the inherent variability induced by stochastic elements of the…
The paper presents some models for the propensity score. Considerable attention is given to a recently popular, but relatively under-explored setting in causal inference where the no-interference assumption does not hold. We lay out some…