Related papers: Signal significance incorporating systematic uncer…
We propose a method to estimate the probability of new physics discovery in future high energy physics experiments. Physics simulation gives both the average numbers $<N_b>$ of background and $<N_s>$ of signal events. We find that the…
In the report the approach to estimation of quality of planned experiments is considered. This approach is based on the analysis of uncertainty, which will take place under the future hypotheses testing about the existence of a new…
We present a procedure for calculating an upper limit on the number of signal events which incorporates the Poisson uncertainty in the background, estimated from control regions of one or two dimensions. For small number of signal events,…
Statistical significance testing of differences in values of metrics like recall, precision and balanced F-score is a necessary part of empirical natural language processing. Unfortunately, we find in a set of experiments that many commonly…
This paper presents a novel approach to estimate the Standard Model backgrounds based on modifying Monte Carlo predictions within their systematic uncertainties. The improved background model is obtained by altering the original predictions…
Several experiments in high-energy physics and astrophysics can be treated as on/off measurements, where an observation potentially containing a new source or effect ("on" measurement) is contrasted with a background-only observation free…
A large fraction of papers in the climate literature includes erroneous uses of significance tests. A Bayesian analysis is presented to highlight the meaning of significance tests and why typical misuse occurs. It is concluded that a…
This article discusses the determination of asymmetries. We consider a sample of events consisting of a peak of signal events on top of some background events. Both signal and background have an unknown asymmetry, e.g. a spin or…
We propose a method to estimate the probability of new physics discovery in future high energy physics experiments. Physics simulation gives both the average numbers $<N_b>$ of background and $<N_s>$ of signal events. We find that the…
The incorporation of systematic uncertainties into confidence interval calculations has been addressed recently in a paper by Conrad et al. (Physical Review D 67 (2003) 012002). In their work, systematic uncertainities in detector…
Continuous monitoring is becoming more popular due to its significant benefits, including reducing sample sizes and reaching earlier conclusions. In general, it involves monitoring nuisance parameters (e.g., the variance of outcomes) until…
We consider a quantum system that is being continuously monitored, giving rise to a measurement signal. From such a stream of data, information needs to be inferred about the underlying system's dynamics. Here we focus on hypothesis testing…
Classifiers are often tested on relatively small data sets, which should lead to uncertain performance metrics. Nevertheless, these metrics are usually taken at face value. We present an approach to quantify the uncertainty of…
Quantitative characterizations and estimations of uncertainty are of fundamental importance in optimization and decision-making processes. Herein, we propose intuitive scores, which we call certainty and doubt, that can be used in both a…
While considerable advance has been made to account for statistical uncertainties in astronomical analyses, systematic instrumental uncertainties have been generally ignored. This can be crucial to a proper interpretation of analysis…
In this article we study the problem of quantifying the uncertainty in an experiment with a technical system. We propose new density estimates which combine observed data of the technical system and simulated data from an (imperfect)…
Unquantified sources of uncertainty in observational causal analyses can break the integrity of the results. One would never want another analyst to repeat a calculation with the same dataset, using a seemingly identical procedure, only to…
In this Reply we include the corrections suggested in the Comment [Phys. Rev. Lett. 131, 169001]. We show that their impact on our results is small, and that the overall conclusion of the Article [Phys. Rev. Lett. 129, 111102] are robust.…
We study the robustness of conformal prediction, a powerful tool for uncertainty quantification, to label noise. Our analysis tackles both regression and classification problems, characterizing when and how it is possible to construct…
In this note, an alternative for presenting the distribution of `significant' events in searches for new phenomena is described. The alternative is based on probability density functions used in the evaluation of the `significance' of an…