Related papers: A simple sketching algorithm for entropy estimatio…
Entropy estimation, due in part to its connection with mutual information, has seen considerable use in the study of time series data including causality detection and information flow. In many cases, the entropy is estimated using…
We consider the problem of sketching the $p$-th frequency moment of a vector, $p>2$, with multiplicative error at most $1\pm \epsilon$ and \emph{with high confidence} $1-\delta$. Despite the long sequence of work on this problem, tight…
Predictive inference requires balancing statistical accuracy against informational complexity, yet the choice of complexity measure is usually imposed rather than derived. We treat econometric objects as predictive rules, mappings from…
This paper considers an information bottleneck problem with the objective of obtaining a most informative representation of a hidden feature subject to a R\'enyi entropy complexity constraint. The optimal bottleneck trade-off between…
We present simple and computationally efficient nonparametric estimators of R\'enyi entropy and mutual information based on an i.i.d. sample drawn from an unknown, absolutely continuous distribution over $\R^d$. The estimators are…
Entropy has been a common index to quantify the complexity of time series in a variety of fields. Here, we introduce increment entropy to measure the complexity of time series in which each increment is mapped into a word of two letters,…
We consider fitting a bivariate spline regression model to data using a weighted least-squares cost function, with weights that sum to one to form a discrete probability distribution. By applying the principle of maximum entropy, the weight…
Conventional information-theoretic quantities assume access to probability distributions. Estimating such distributions is not trivial. Here, we consider function-based formulations of cross entropy that sidesteps this a priori estimation…
We describe an approach to improving model fitting and model generalization that considers the entropy of distributions of modelling residuals. We use simple simulations to demonstrate the observational signatures of overfitting on ordered…
We introduce a new computational model for data streams: asymptotically exact streaming algorithms. These algorithms have an approximation ratio that tends to one as the length of the stream goes to infinity while the memory used by the…
This paper considers the entropy of the sum of (possibly dependent and non-identically distributed) Bernoulli random variables. Upper bounds on the error that follows from an approximation of this entropy by the entropy of a Poisson random…
The R\'enyi-Shannon entropy associated to critical quantum spins chain with central charge $c=1$ is shown to have a phase transition at some value $n_c$ of the R\'enyi parameter $n$ which depends on the Luttinger parameter (or…
The matrix-based Renyi's \alpha-entropy functional and its multivariate extension were recently developed in terms of the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS).…
We study the estimation of Tsallis entropy of a finite number of independent populations, each following an exponential distribution with the same scale parameter and distinct location parameters for $q>0$. We derive a Stein-type improved…
We study the computation of the Petz-Augustin mean of order $\alpha \in (0,1) \cup (1,\infty)$, defined as the minimizer of a weighted sum of $n$ Petz-R\'enyi divergences of order $\alpha$ over the set of $d$-by-$d$ quantum states, where…
One of the most influential results in neural network theory is the universal approximation theorem [1, 2, 3] which states that continuous functions can be approximated to within arbitrary accuracy by single-hidden-layer feedforward neural…
We investigate the performance of entropy estimation methods, based either on block entropies or compression approaches, in the case of bidimensional sequences. We introduce a validation dataset made of images produced by a large number of…
Test-time adaptation aims to adapt to realistic environments in an online manner by learning during test time. Entropy minimization has emerged as a principal strategy for test-time adaptation due to its efficiency and adaptability.…
Entropy-type integral functionals of densities are widely used in mathematical statistics, information theory, and computer science. Examples include measures of closeness between distributions (e.g., density power divergence) and…
The entropy of probability distribution defined by Shannon has several extensions. R\'enyi entropy is one of the general extensions of Shannon entropy and is widely used in engineering, physics, and so on. On the other hand, the quantum…