Related papers: Relative Information Loss - An Introduction
Information theory plays a central role in establishing fundamental limits on what any learning or estimation algorithm can -- and cannot -- achieve, regardless of computational power. In this chapter, we provide an introduction to these…
Finite abstractions are discrete approximations of dynamical systems, such that the set of abstraction trajectories contains all system trajectories. There is a consensus that abstractions suffer from the curse of dimensionality: for the…
Entropy and information provide natural measures of correlation among elements in a network. We construct here the information theoretic analog of connected correlation functions: irreducible $N$--point correlation is measured by a decrease…
Rooted trees with probabilities are used to analyze properties of a variable length code. A bound is derived on the difference between the entropy rates of the code and a memoryless source. The bound is in terms of normalized informational…
Missing values arise in most real-world data sets due to the aggregation of multiple sources and intrinsically missing information (sensor failure, unanswered questions in surveys...). In fact, the very nature of missing values usually…
In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One…
In [Berta et al., J. Math. Phys. 56, 022205 (2015)], we recently proposed Renyi generalizations of the conditional quantum mutual information of a tripartite state on $ABC$ (with $C$ being the conditioning system), which were shown to…
The hallmark of deterministic chaos is that it creates information---the rate being given by the Kolmogorov-Sinai metric entropy. Since its introduction half a century ago, the metric entropy has been used as a unitary quantity to measure a…
We analyze the problem of sequential probability assignment for binary outcomes with side information and logarithmic loss, where regret---or, redundancy---is measured with respect to a (possibly infinite) class of experts. We provide upper…
We extend the duality between exponential integrals and relative entropy to a variational formula for exponential integrals involving the Renyi divergence. This formula characterizes the dependence of risk-sensitive functionals and related…
We introduce one-way unlocalizable information deficit with respect to the one-way information deficit, similar to the definition of one-way unlocalizable quantum discord with respect to one-way quantum discord. The properties of the…
Selecting a minimal feature set that is maximally informative about a target variable is a central task in machine learning and statistics. Information theory provides a powerful framework for formulating feature selection algorithms --…
We study a dynamical system with time dependent Hamiltonian by numerical experiments so as to find a relation between thermodynamics and chaotic nature of the system. Excess information loss, defined newly based on Lyapunov analysis, is…
The problem of guessing a random string is revisited. A close relation between guessing and compression is first established. Then it is shown that if the sequence of distributions of the information spectrum satisfies the large deviation…
Super-resolution as an ill-posed problem has many high-resolution candidates for a low-resolution input. However, the popular $\ell_1$ loss used to best fit the given HR image fails to consider this fundamental property of non-uniqueness in…
Forecasting is usually framed as a problem of model choice. This paper starts earlier, asking how much predictive information is available at each horizon. Under logarithmic loss, the answer is exact: the mutual information between the…
Information functionals allow to quantify the degree of randomness of a given probability distribution, either absolutely (through min/max entropy principles) or relative to a prescribed reference one. Our primary aim is to analyze the…
The minimum rate needed to accurately approximate a product distribution based on an unnormalized informational divergence is shown to be a mutual information. This result subsumes results of Wyner on common information and Han-Verd\'{u} on…
Information theory on a time-discrete setting in the framework of time series analysis is generalized to the time-continuous case. Considerations of the Roessler and Lorenz dynamics as well as the Ornstein-Uhlenbeck process yield for…
We consider approximation or recovery of functions based on a finite number of function evaluations. This is a well-studied problem in optimal recovery, machine learning, and numerical analysis in general, but many fundamental insights were…