Related papers: Tighter Confidence Intervals for Rating Systems
Prediction credibility measures, in the form of confidence intervals or probability distributions, are fundamental in statistics and machine learning to characterize model robustness, detect out-of-distribution samples (outliers), and…
AB-testing is a very popular technique in web companies since it makes it possible to accurately predict the impact of a modification with the simplicity of a random split across users. One of the critical aspects of an AB-test is its…
The System Usability Scale (SUS) is a short, survey-based approach used to determine the usability of a system from an end user perspective once a prototype is available for assessment. Individual scores are gathered using a 10-question…
Certifiable, adaptive uncertainty estimates for unknown quantities are an essential ingredient of sequential decision-making algorithms. Standard approaches rely on problem-dependent concentration results and are limited to a specific…
In recent machine learning systems, confidence scores are being utilized more and more to manage selective prediction, whereby a model can abstain from making a prediction when it is unconfident. Yet, conventional metrics like accuracy,…
We consider the power to reject false values of the parameter in Frequentist methods for the calculation of confidence intervals. We connect the power with the physical significance (reliability) of confidence intervals for a parameter…
The following questions are discussed: ``Why confidence intervals are a hot topic?''; ``Are confidence intervals objective?''; ``What is the usefulness of coverage?''; ``How to obtain useful information from experiment?''; ``The confidence…
Experimental designs are tools which can drastically reduce the number of simulations required by time-consuming computer codes. One strategy for selecting the values of the inputs, whose response is to be observed, is to choose these…
Modern software systems are typically configurable, a fundamental prerequisite for wide applicability and reusability. This flexibility poses an extraordinary challenge for quality assurance, as the enormous number of possible…
Prediction intervals are commonly used in meta-analysis with random-effects models. One widely used method, the Higgins-Thompson-Spiegelhalter prediction interval, replaces the heterogeneity parameter with its point estimate, but its…
Calibration, the practice of choosing the parameters of a structural model to match certain empirical moments, can be viewed as minimum distance estimation. Existing standard error formulas for such estimators require a consistent estimate…
Density-based directed distances -- particularly known as divergences -- between probability distributions are widely used in statistics as well as in the adjacent research fields of information theory, artificial intelligence and machine…
Model evolution and constant availability of data are two common phenomena in large-scale real-world machine learning applications, e.g. ads and recommendation systems. To adapt, the real-world system typically retrain with all available…
Using normal approximation (NA) to construct a kernel-smoother-based confidence interval faces a fundamental challenge: the normalization makes a small estimation bias become a non-negligible inferential bias. This paper takes a different…
We provide adaptive confidence intervals on a parameter of interest in the presence of nuisance parameters when some of the nuisance parameters have known signs. The confidence intervals are adaptive in the sense that they tend to be short…
Quantum state tomography is the standard technique for reconstructing a quantum state from experimental data. In the regime of finite statistics, experimental data cannot give perfect information about the quantum state. A common way to…
Confidence intervals for the means of multiple normal populations are often based on a hierarchical normal model. While commonly used interval procedures based on such a model have the nominal coverage rate on average across a population of…
We introduce a new framework for creating point-wise confidence intervals for the distribution of event times for current status data. Existing methods are based on asymptotics. Our framework is based on binomial properties and motivates…
We study confidence intervals based on hard-thresholding, soft-thresholding, and adaptive soft-thresholding in a linear regression model where the number of regressors $k$ may depend on and diverge with sample size $n$. In addition to the…
Classical machine learning classifiers tend to be overconfident can be unreliable outside of the laboratory benchmarks. Properly assessing the reliability of the output of the model per sample is instrumental for real-life scenarios where…