Related papers: $H(X)$ vs. $H(f(X))$
The maximum entropy approach operating with quite general entropy measure and constraint is considered. It is demonstrated that for a conditional or parametrized probability distribution $f(x|\mu)$ there is a "universal" relation among the…
Given a continuous real-valued function on [0, 1], and a closed subset E \subset [0, 1] we denote by f E the restriction of f to E, that is, the function defined only on E that takes the same values as f at every point of E >. The…
Let $X$ be a Bernoulli random variable with the success probability $p$. We are interested in tight bounds on $\mathbb{E}[f(X_1,X_2)]$, where $X_i=\mathbb{E}[X| \mathcal{F}_i]$ and $\mathcal{F}_i$ are some sigma-algebras. This problem is…
``Behind every limit theorem, there is an inequality'' said Kolmogorov. We say ``for every inequality, there is an approximate inequality under approximate regularity conditions.'' Suppose $X, X'$ are independent and identically distributed…
The relationship between three probability distributions and their maximizable entropy forms is discussed without postulating entropy property. For this purpose, the entropy I is defined as a measure of uncertainty of the probability…
We provide an upper bound as a random variable for the functions of estimators in high dimensions. This upper bound may help establish the rate of convergence of functions in high dimensions. The upper bound random variable may converge…
The universal bound on specific entropy was originally inferred from black hole thermodynamics. We here show from classical thermodynamics alone that for a system at fixed volume or fixed pressure, the ratio of entropy to nonrelativistic…
We define a one-parameter family of entropies, each assigning a real number to any probability measure on a compact metric space (or, more generally, a compact Hausdorff space with a notion of similarity between points). These entropies…
This paper derives new bounds on the difference of the entropies of two discrete random variables in terms of the local and total variation distances between their probability mass functions. The derivation of the bounds relies on maximal…
A new upper bound on the relative entropy is derived as a function of the total variation distance for probability measures defined on a common finite alphabet. The bound improves a previously reported bound by Csisz\'ar and Talata. It is…
We present a unifying approach to the study of entropies in Mathematics, such as measure entropy, topological entropy, algebraic entropy, set-theoretic entropy. We take into account discrete dynamical systems, that is, pairs $(X,T)$, where…
We investigate the properties of a discrete-time martingale $\{X_m\}_{m\in \mathbb Z_{\geq 0}}$, where all differences between adjacent random variables are limited to be not more than a constant as a promise. In this situation, it is known…
We provide a systematic approach to deal with the following problem. Let $X_1,\ldots,X_n$ be, possibly dependent, $[0,1]$-valued random variables. What is a sharp upper bound on the probability that their sum is significantly larger than…
Suppose $X$ is a uniformly distributed $n$-dimensional binary vector and $Y$ is obtained by passing $X$ through a binary symmetric channel with crossover probability $\alpha$. A recent conjecture by Courtade and Kumar postulates that…
Consider a discrete-time martingale $\{X_t\}$ taking values in a Hilbert space $\mathcal H$. We show that if for some $L \geq 1$, the bounds $\mathbb{E} \left[\|X_{t+1}-X_t\|_{\mathcal H}^2 \mid X_t\right]=1$ and $\|X_{t+1}-X_t\|_{\mathcal…
We derive an upper bound for the mean of the supremum of the empirical process indexed by a class of functions that are known to have variance bounded by a small constant $\delta$. The bound is expressed in the uniform entropy integral of…
In this paper, we focus on some properties, calculations and estimations of topological entropy for a nonautonomous dynamical system $(X,f_{0,\infty})$ generated by a sequence of continuous self-maps $f_{0,\infty}=\{f_n\}_{n=0}^{\infty}$ on…
An important tool to quantify the likeness of two probability measures are f-divergences, which have seen widespread application in statistics and information theory. An example is the total variation, which plays an exceptional role among…
We study the multiplicative version of the classical Furstenberg's filtering problem, where instead of the sum $\mathbf{X}+\mathbf{Y}$ one considers the product $\mathbf{X}\cdot \mathbf{Y}$ ($\mathbf{X}$ and $\mathbf{Y}$ are bilateral,…
One of the difficulties in calculating the capacity of certain Poisson channels is that H(lambda), the entropy of the Poisson distribution with mean lambda, is not available in a simple form. In this work we derive upper and lower bounds…