Related papers: P values, confidence intervals, or confidence leve…
Confidence estimation (CE) indicates how reliable the answers of large language models are and impacts user trust and decision-making. Existing evaluations mainly concern the alignment between confidence and correctness, but ignore the…
Incorrect usage of $p$-values, particularly within the context of significance testing using the arbitrary .05 threshold, has become a major problem in modern statistical practice. The prevalence of this problem can be traced back to the…
A confidence distribution is a complete tool for making frequentist inference for a parameter of interest $\psi$ based on an assumed parametric model. Indeed, it allows to reach point estimates, to assess their precision, to set up tests…
The usual procedure for estimating the significance of a peak in a power spectrum is to calculate the probability of obtaining that value or a larger value by chance (known as the "p-value"), on the assumption that the time series contains…
We marshall the arguments for preferring Bayesian hypothesis testing and confidence sets to frequentist ones. We define admissible solutions to inference problems, noting that Bayesian solutions are admissible. We give seven weaker…
In this paper, we present a new classifier, which integrates significance testing results over different random subspaces to yield consensus p-values for quantifying the uncertainty of classification decision. The null hypothesis is that…
The main purpose of this paper is to introduce first a new family of empirical test statistics for testing a simple null hypothesis when the vector of parameters of interest are defined through a specific set of unbiased estimating…
We study the frequentist properties of confidence intervals computed by the method known to statisticians as the Profile Likelihood. It is seen that the coverage of these intervals is surprisingly good over a wide range of possible…
This article describes an efficient procedure for computing approximate confidence levels for searches for new particles where the expected signal and background levels are small enough to require the use of Poisson statistics. The results…
Effective software safety standards will contribute to confidence, or assurance, in the safety of the systems in which the software is used. It is infeasible to demonstrate a correlation between standards and accidents, but there is an…
Investigators often use the data to generate interesting hypotheses and then perform inference for the generated hypotheses. P-values and confidence intervals must account for this explorative data analysis. A fruitful method for doing so…
The following zero-sum game between nature and a statistician blends Bayesian methods with frequentist methods such as p-values and confidence intervals. Nature chooses a posterior distribution consistent with a set of possible priors. At…
A new formulation for the proportion of true null hypotheses $(\pi_0)$, based on the sum of all $p$-values and the average of expected $p$-value under the false null hypotheses has been proposed in the current work. This formulation of the…
In this article, we derive an explicit formula for computing confidence interval for the mean of a bounded random variable. Moreover, we have developed multistage point estimation methods for estimating the mean value with prescribed…
In Neyman's original formulation, a 1-alpha confidence interval procedure is justified by its long-run coverage properties, and a single realized interval is to be described only by the slogan that it either covers the parameter or it does…
A confidence measure is able to estimate the reliability of an hypothesis provided by a machine translation system. The problem of confidence measure can be seen as a process of testing : we want to decide whether the most probable sequence…
Language models (LMs) should provide reliable confidence estimates to help users detect mistakes in their outputs and defer to human experts when necessary. Asking a language model to assess its confidence ("Score your confidence from…
Probability intervals are an attractive tool for reasoning under uncertainty. Unlike belief functions, though, they lack a natural probability transformation to be used for decision making in a utility theory framework. In this paper we…
Probability forecasts for binary events play a central role in many applications. Their quality is commonly assessed with proper scoring rules, which assign forecasts a numerical score such that a correct forecast achieves a minimal…
P-hacking poses challenges to traditional hypothesis testing. In this paper, we propose a robust method for the one-sample significance test that can protect against p-hacking from sample manipulation. Precisely, assuming a sequential…