Related papers: Finite-Sample Concentration of the Multinomial in …
We study conditions under which integer sequences with independent, identically distributed gaps are asymptotically $k$-complete, meaning that every sufficiently large integer can be represented as the sum of exactly $k$ distinct elements…
The Kullback-Leibler (KL) divergence is a foundational measure for comparing probability distributions. Yet in multivariate settings, its single value often obscures the underlying reasons for divergence, conflating mismatches in individual…
Cross-entropy loss is a common choice when it comes to multiclass classification tasks and language modeling in particular. Minimizing this loss results in language models of very good quality. We show that it is possible to fine-tune these…
A finite form of de Finetti's representation theorem is established using elementary information-theoretic tools: The distribution of the first $k$ random variables in an exchangeable binary vector of length $n\geq k$ is close to a mixture…
We initiate a study of the following problem: Given a continuous domain $\Omega$ along with its convex hull $\mathcal{K}$, a point $A \in \mathcal{K}$ and a prior measure $\mu$ on $\Omega$, find the probability density over $\Omega$ whose…
This paper shows that large nonparametric classes of conditional multivariate densities can be approximated in the Kullback--Leibler distance by different specifications of finite mixtures of normal regressions in which normal means and…
The measure-theoretic definition of Kullback-Leibler relative-entropy (KL-entropy) plays a basic role in the definitions of classical information measures. Entropy, mutual information and conditional forms of entropy can be expressed in…
In this note we generalize the results of the recent work by Tom Bohman and Jacob Hofstad on the independence number in G(n, p) to the case of the random k-uniform hypergraph. Concentration in two values occurs in the regime…
We present novel bounds for estimating discrete probability distributions under the $\ell_\infty$ norm. These are nearly optimal in various precise senses, including a kind of instance-optimality. Our data-dependent convergence guarantees…
The following anticoncentration property is proved. The probability that the $k$-order statistic of an arbitrarily correlated jointly Gaussian random vector $X$ with unit variance components lies within an interval of length $\varepsilon$…
In \cite{FTD1}, we proved the almost sure convergence of eigenvalues of the SYK model, which can be viewed as a type of \emph{law of large numbers} in probability theory; in \cite{FTD2}, we proved that the linear statistic of eigenvalues…
We introduce a new generalization of relative entropy to non-negative vectors with sums $\gt 1$. We show in a purely combinatorial setting, with no probabilistic considerations, that in the presence of linear constraints defining a convex…
Generative models frequently suffer miscalibration, wherein statistics of the sampling distribution, such as the fraction of generations in a given class, deviate from desired values. We frame calibration as a constrained optimization…
The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning. In this paper, we analyze estimates of differential entropy in $K$-dimensional Euclidean space,…
The goal of importance sampling is to estimate the expected value of a given function with respect to a probability measure $\nu$ using a random sample of size $n$ drawn from a different probability measure $\mu$. If the two measures $\mu$…
Estimating the Shannon entropy of a discrete distribution from which we have only observed a small sample is challenging. Estimating other information-theoretic metrics, such as the Kullback-Leibler divergence between two sparsely sampled…
We establish effective convergence rates in the Doeblin-Lenstra law, describing the limiting distribution of approximation coefficients arising from continued fraction convergents of a typical real number. More generally, we prove…
We study the problem of generating a random variate $X$ from a finite discrete probability distribution $P$ using an entropy source of independent fair coin flips. A classic result from Knuth and Yao shows that the optimal expected number…
We deal with countable alphabet locally compact random subshifts of finite type (the latter merely meaning that the symbol space is generated by an incidence matrix) under the absence of Big Images Property and under the absence of uniform…
Finite heat reservoir capacity and temperature fluctuations lead to modification of the well known canonical exponential weight factor. Requiring that the corrections least depend on the one-particle energy, we derive a deformed entropy,…