Related papers: Relative Information Loss - An Introduction
Current discrete randomness and information conservation inequalities are over total recursive functions, i.e. restricted to deterministic processing. This restriction implies that an algorithm can break algorithmic randomness conservation…
We introduce an axiomatic approach to entropies and relative entropies that relies only on minimal information-theoretic axioms, namely monotonicity under mixing and data-processing as well as additivity for product distributions. We find…
We define a measure of redundant information based on projections in the space of probability distributions. Redundant information between random variables is information that is shared between those variables. But in contrast to mutual…
A new finite form of de Finetti's representation theorem is established using elementary information-theoretic tools. The distribution of the first $k$ random variables in an exchangeable vector of $n\geq k$ random variables is close to a…
This dissertation investigates relative entropies, also called generalized divergences, and how they can be used to characterize information-theoretic tasks in quantum information theory. The main goal is to further refine characterizations…
The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some of the key properties. We show that this can be solved by adding a regularization…
We introduce notions of information/entropy and information loss associated to exponentiable motivic measures. We show that they satisfy appropriate analogs to the Khinchin-type properties that characterize information loss in the context…
Our capacity to process information depends on the computational power at our disposal. Information theory captures our ability to distinguish states or communicate messages when it is unconstrained with unrivaled beauty and elegance. For…
Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss. This puzzling contradiction necessitates new approaches to the study of overfitting. Here we quantify…
This paper discusses the thermodynamic irreversibility realized in high-dimensional Hamiltonian systems with a time-dependent parameter. A new quantity, the irreversible information loss, is defined from the Lyapunov analysis so as to…
High dimensional data can have a surprising property: pairs of data points may be easily separated from each other, or even from arbitrary subsets, with high probability using just simple linear classifiers. However, this is more of a rule…
Sequential decision-making systems routinely operate with missing or incomplete data. Classical reinforcement learning theory, which is commonly used to solve sequential decision problems, assumes Markovian observability, which may not hold…
A new upper bound on the relative entropy is derived as a function of the total variation distance for probability measures defined on a common finite alphabet. The bound improves a previously reported bound by Csisz\'ar and Talata. It is…
We consider a natural measure of relevance: the reduction in optimal prediction risk in the presence of side information. For any given loss function, this relevance measure captures the benefit of side information for performing inference…
It is impossible to recover a vector from $\mathbb{R}^m$ with less than $m$ linear measurements, even if the measurements are chosen adaptively. Recently, it has been shown that one can recover vectors from $\mathbb{R}^m$ with arbitrary…
Information and uncertainty are closely related and extensively studied concepts in a number of scientific disciplines such as communication theory, probability theory, and statistics. Increasing the information arguably reduces the…
We study approximation and integration problems and compare the quality of optimal information with the quality of random information. For some problems random information is almost optimal and for some other problems random information is…
The study of a machine learning problem is in many ways is difficult to separate from the study of the loss function being used. One avenue of inquiry has been to look at these loss functions in terms of their properties as scoring rules…
This paper considers the problem of defining a measure of redundant information that quantifies how much common information two or more random variables specify about a target random variable. We discussed desired properties of such a…
Given finite-dimensional random vectors $Y$, $X$, and $Z$ that form a Markov chain in that order (i.e., $Y \to X \to Z$), we derive upper bounds on the excess minimum risk using generalized information divergence measures. Here, $Y$ is a…