Related papers: A New Estimator of Kullback--Leibler Divergence vi…
We interpret likelihood-based test functions from a geometric perspective where the Kullback-Leibler (KL) divergence is adopted to quantify the distance from a distribution to another. Such a test function can be seen as a sub-Gaussian…
The problem of estimating the Kullback-Leibler divergence $D(P\|Q)$ between two unknown distributions $P$ and $Q$ is studied, under the assumption that the alphabet size $k$ of the distributions can scale to infinity. The estimation is…
Kullback-Leibler (KL) divergence is a fundamental concept in information theory that quantifies the discrepancy between two probability distributions. In the context of Variational Autoencoders (VAEs), it serves as a central regularization…
With the high-precision data from current and upcoming experiments, it becomes increasingly important to perform consistency tests of the standard cosmological model. In this work, we focus on consistency measures between different data…
For generic systems exhibiting power law behaviors, and hence multiscale dependencies, we propose a new, and yet simple, tool to analyze multifractality and intermittency, after noticing that these concepts are directly related to the…
We consider an extension of $\epsilon$-entropy to a KL-divergence based complexity measure for randomized density estimation methods. Based on this extension, we develop a general information-theoretical inequality that measures the…
We analyze dynamical properties of the Shannon information entropy of a continuous probability distribution, which is driven by a standard diffusion process. This entropy choice is confronted with another option, employing the conditional…
We study the problem of closeness testing for continuous distributions and its implications for causal discovery. Specifically, we analyze the sample complexity of distinguishing whether two multidimensional continuous distributions are…
The Kullback-Leibler (KL) divergence is frequently used in data science. For discrete distributions on large state spaces, approximations of probability vectors may result in a few small negative entries, rendering the KL divergence…
Data collection is a critical step in statistical inference and data science, and the goal of statistical experimental design (ED) is to find the data collection setup that can provide most information for the inference. In this work we…
Universal hypothesis testing refers to the problem of deciding whether samples come from a nominal distribution or an unknown distribution that is different from the nominal distribution. Hoeffding's test, whose test statistic is equivalent…
The entropy is one of the most applicable uncertainty measures in many statistical and en- gineering problems. In statistical literature, the entropy is used in calculation of the Kullback- Leibler (KL) information which is a powerful mean…
The Kullback-Leibler (KL) divergence is a fundamental equation of information theory that quantifies the proximity of two probability distributions. Although difficult to understand by examining the equation, an intuition and understanding…
Experimental designs are tools which can dramatically reduce the number of simulations required by time-consuming computer codes. Because we don't know the true relation between the response and inputs, designs should allow one to fit a…
Estimating the Kullback--Leibler (KL) divergence between language models has many applications, e.g., reinforcement learning from human feedback (RLHF), interpretability, and knowledge distillation. However, computing the exact KL…
The maximum entropy principle is a powerful tool for solving underdetermined inverse problems. This paper considers the problem of discretizing a continuous distribution, which arises in various applied fields. We obtain the approximating…
The Kullback-Leibler divergence or relative entropy is an information-theoretic measure between statistical models that play an important role in measuring a distance between random variables. In the study of complex systems, random fields…
This work explores connections between the quantum relative entropy of two faithful states $\rho,\sigma$ (i.e. full-rank density matrices) and the Kullback-Leibler divergences of classical measures $\mu,\nu$. Here, $\mu$ and $\nu$ are…
This paper mainly focuses on studying the Shannon Entropy and Kullback-Leibler divergence of the multivariate log canonical fundamental skew-normal (LCFUSN) and canonical fundamental skew-normal (CFUSN) families of distributions, extending…
The Kullback-Leibler (KL) divergence is a foundational measure for comparing probability distributions. Yet in multivariate settings, its single value often obscures the underlying reasons for divergence, conflating mismatches in individual…