Related papers: Are you using test log-likelihood correctly?
Modern statisticians are often presented with hundreds or thousands of hypothesis testing problems to evaluate at the same time, generated from new scientific technologies such as microarrays, medical and satellite imaging devices, or flow…
Large language models (LLMs) are a type of machine learning tool that economists have started to apply in their empirical research. One such application is macroeconomic forecasting with backtesting of LLMs, even though they are trained on…
Statistical significance testing of differences in values of metrics like recall, precision and balanced F-score is a necessary part of empirical natural language processing. Unfortunately, we find in a set of experiments that many commonly…
The value of the likelihood is occasionally used by high energy physicists as a statistic to measure goodness-of-fit in unbinned maximum likelihood fits. Simple examples are presented that illustrate why this (seemingly intuitive) method…
The standard loss functions used in the literature on probabilistic prediction are the log loss function, the Brier loss function, and the spherical loss function; however, any computable proper loss function can be used for comparison of…
The estimated accuracy of a classifier is a random quantity with variability. A common practice in supervised machine learning, is thus to test if the estimated accuracy is significantly better than chance level. This method of signal…
Comparing competing mathematical models of complex natural processes is a shared goal among many branches of science. The Bayesian probabilistic framework offers a principled way to perform model comparison and extract useful metrics for…
The log-transform is a common tool in statistical analysis, reducing the impact of extreme values, compressing the range of reported values for improved visualization, enabling the usage of parametric statistical tests requiring normally…
Likelihood functions are ubiquitous in data analyses at the LHC and elsewhere in particle physics. Partly because "probability" and "likelihood" are virtual synonyms in everyday English, but crucially distinct in data analysis, there is…
Bayesian hypothesis testing is investigated when the prior probabilities of the hypotheses, taken as a random vector, are quantized. Nearest neighbor and centroid conditions are derived using mean Bayes risk error as a distortion measure…
It is quite common in modern research, for a researcher to test many hypotheses. The statistical (frequentist) hypothesis testing framework, does not scale with the number of hypotheses in the sense that naively performing many hypothesis…
We report an inconsistency found in probability theory (also referred to as measure-theoretic probability). For probability measures induced by real-valued random variables, we deduce an "equality" such that one side of the "equality" is a…
In this paper we show that there is a link between approximate Bayesian methods and prior robustness. We show that what is typically recognized as an approximation to the likelihood, either due to the simulated data as in the Approximate…
Motivated by applications to goodness of fit testing, the empirical likelihood approach is generalized to allow for the number of constraints to grow with the sample size and for the constraints to use estimated criteria functions. The…
This paper studies the problem of testing whether a system of linear equality and inequality constraints admits a solution when the coefficients of that system may have to be estimated. We show that a wide range of inferential questions in…
Context: It has been argued that software engineering replications are useful for verifying the results of previous experiments. However, it has not yet been agreed how to check whether the results hold across replications. Besides, some…
Non-Gaussian likelihoods are essential for modelling complex real-world observations but pose significant computational challenges in learning and inference. Even with Gaussian priors, non-Gaussian likelihoods often lead to analytically…
The goodness of the long-term prediction in the state-space model was evaluated using the squared long-term prediction error. In order to estimate the model parameters suitable for long-term prediction, we devised a modified log-likelihood…
Mixed linear models are commonly used in repeated measures studies. They account for the dependence amongst observations obtained from the same experimental unit. Oftentimes, the number of observations is small, and it is thus important to…
Log-Euclidean distances are commonly used to quantify the similarity between positive definite matrices using geometric considerations. This paper analyzes the behavior of this distance when it is used to measure closeness between…