Related papers: Asymptotically Optimal Massey-Like Inequality on G…
Algorithmic entropy and Shannon entropy are two conceptually different information measures, as the former is based on size of programs and the later in probability distributions. However, it is known that, for any recursive probability…
Subword tokenization is a key part of many NLP pipelines. However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performance than others. We propose that good tokenizers lead to…
This paper is on developing some computer-assisted proof methods involving non-classical inequalities for Shannon entropy. Two areas of the applications of information inequalities are studied: Secret sharing schemes and hat guessing games.…
Learning the minimum/maximum mean among a finite set of distributions is a fundamental sub-task in planning, game tree search and reinforcement learning. We formalize this learning task as the problem of sequentially testing how the minimum…
We study the notion of approximate entropy within the framework of network theory. Approximate entropy is an uncertainty measure originally proposed in the context of dynamical systems and time series. We firstly define a purely structural…
We prove the upper semicontinuity of the measure theoretic entropy for the geodesic flow on complete Riemannian manifolds without focal points and bounded sectional curvature. We then study the relationship between the escape of mass…
From the output produced by a memoryless deletion channel from a uniformly random input of known length $n$, one obtains a posterior distribution on the channel input. The difference between the Shannon entropy of this distribution and that…
We present a technique for entropy optimization to calculate a distribution from its moments. The technique is based upon maximizing a discretized form of the Shannon entropy functional by mapping the problem onto a dual space where an…
We show that in the continuum limit, the average spectrum method (ASM) is equivalent to maximizing R\'enyi entropies of order $\eta$, of which Shannon entropy is the special case $\eta=1$. The order of R\'enyi entropy is determined by the…
Expressions for (EPI Shannon type) Divergence-Power Inequalities (DPI) in two cases (time-discrete and band-limited time-continuous) of stationary random processes are given. The new expressions connect the divergence rate of the sum of…
Gaussian mixture distributions are commonly employed to represent general probability distributions. Despite the importance of using Gaussian mixtures for uncertainty estimation, the entropy of a Gaussian mixture cannot be calculated…
Fermionic Gaussian states are a fundamental tool in many-body physics, faithfully representing non-interacting quantum systems and allowing for efficient numerical simulations. Given a many-body wave function, it is therefore interesting to…
The Shannon entropy is used as a basis for applying different lemmas and conjectures concerning the set of gaps between prime numbers G_p , thus estimating several measures of it. The same procedures are applied to artificially created…
This paper considers an information bottleneck problem with the objective of obtaining a most informative representation of a hidden feature subject to a R\'enyi entropy complexity constraint. The optimal bottleneck trade-off between…
A guessing wiretapper's performance on a Shannon cipher system is analyzed for a source with memory. Close relationships between guessing functions and length functions are first established. Subsequently, asymptotically optimal encryption…
We study a quantity called discrete layered entropy, which approximates the Shannon entropy within a logarithmic gap. Compared to the Shannon entropy, the discrete layered entropy is piecewise linear, approximates the expected length of the…
We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to…
New non-asymptotic random coding theorems (with error probability $\epsilon$ and finite block length $n$) based on Gallager parity check ensemble and Shannon random code ensemble with a fixed codeword type are established for discrete input…
Learning directionality between variables is crucial yet challenging, especially for mechanistic relationships without a priori ordering assumptions. We propose a coefficient of asymmetry to quantify directional asymmetry using Shannon's…
In this paper, we prove the anisotropic Shannon inequality for the Renyi entropy with the best constant on Folland-Stein homogeneous Lie groups. As a consequence, we also prove the optimal Shannon inequality in the same setting. Using a…