Related papers: Discrete Layered Entropy, Conditional Compression …

Maximum entropy estimation is of broad interest for inferring properties of systems across many different disciplines. In this work, we significantly extend a technique we previously introduced for estimating the maximum entropy of a set of…

A novel definition of the conditional smooth Renyi entropy, which is different from that of Renner and Wolf, is introduced. It is shown that our definition of the conditional smooth Renyi entropy is appropriate to give lower and upper…

Let $\mathsf{N}_{\rm d}\left[X\right]=\frac{1}{2\pi {\rm e}}{\rm e}^{2\mathsf{H}\left[X\right]}$ denote the entropy power of the discrete random variable $X$ where $\mathsf{H}\left[X\right]$ denotes the discrete entropy of $X$. In this…

Pretrained molecular encoders have become indispensable in computational chemistry for tasks such as property prediction and molecular generation. However, the standard practice of relying solely on final-layer embeddings for downstream…

Assessing whether two datasets are distributionally consistent is central to modern scientific analysis, particularly as generative artificial intelligence produces synthetic data whose fidelity must be validated against real observations…

Deep neural networks tend to exhibit a bias toward low-rank solutions during training, implicitly learning low-dimensional feature representations. This paper investigates how deep multilayer perceptrons (MLPs) encode these feature…

Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as…

Shannon Entropy is the preeminent tool for measuring the level of uncertainty (and conversely, information content) in a random variable. In the field of communications, entropy can be used to express the information content of given…

We derive asymptotic formulas for the number of integer partitions with given sums of $j$th powers of the parts for $j$ belonging to a finite, non-empty set $J \subset \mathbb N$. The method we use is based on the `principle of maximum…

The paper examines relationships between the conditional Shannon entropy and the expectation of $\ell_{\alpha}$-norm for joint probability distributions. More precisely, we investigate the tight bounds of the expectation of…

Differential entropy and log determinant of the covariance matrix of a multivariate Gaussian distribution have many applications in coding, communications, signal processing and statistical inference. In this paper we consider in the high…

Recently, many neural network-based image compression methods have shown promising results superior to the existing tool-based conventional codecs. However, most of them are often trained as separate models for different target bit rates,…

We extend the notion of estimation entropy of autonomous dynamical systems proposed by Liberzon and Mitra [1] to nonlinear dynamical systems with uncertain inputs with bounded variation. We call this new notion the {$\epsilon$}-estimation…

Entropies must correspond to mean values for them to be measurable. The Shannon entropy corresponds to the weighted arithmetic mean, whereas the Renyi entropy corresponds to the exponential mean. These means refer to code lengths, which are…

Statistical mechanics is applied to lossy compression using multilayer perceptrons for unbiased Boolean messages. We utilize a tree-like committee machine (committee tree) and tree-like parity machine (parity tree) whose transfer functions…

Similarity-sensitive entropy measures the uncertainty of a probability law relative to a similarity kernel that encodes the distinguishability between states. We develop a measure-theoretic treatment covering both finite similarity matrices…

In this note, the following basic question is explored: in a cyclic group, how are the Shannon entropies of the sum and difference of i.i.d. random variables related to each other? For the integer group, we show that they can differ by any…

Uniform continuity bounds on entropies are generally expressed in terms of a single distance measure between a pair of probability distributions or quantum states, typically, the total variation distance or trace distance. However, if an…

We consider the problem of learned transform compression where we learn both, the transform as well as the probability distribution over the discrete codes. We utilize a soft relaxation of the quantization operation to allow for…

We illustrate how computer-aided methods can be used to investigate the fundamental limits of the caching systems, which are significantly different from the conventional analytical approach usually seen in the information theory…