Related papers: Relative Information Loss - An Introduction
Finite-state dimension (Dai, Lathrop, Lutz, and Mayordomo (2004)) quantifies the information rate in an infinite sequence as measured by finite-state automata. In this paper, we define a relative version of finite-state dimension. The…
We consider on-line density estimation with a parameterized density from the exponential family. The on-line algorithm receives one example at a time and maintains a parameter that is essentially an average of the past examples. After…
Learning disentangled representations of textual data is essential for many natural language tasks such as fair classification, style transfer and sentence generation, among others. The existent dominant approaches in the context of text…
We investigate the information distribution among different entities in the weak measurements protocol. Focusing on multilevel, decaying systems under continuous (no-click) monitoring, we derive exact, conservation-type information…
For several styles of fidelity constraints -- guaranteed distortion, conditional excess distortion, excess distortion -- we show mutual information upper bounds on the minimum expected description length needed to represent a random…
This work proposes a new loss function targeting classification problems, utilizing a source of information overlooked by cross entropy loss. First, we derive a series of the tightest upper and lower bounds for the probability of a random…
Algorithms in machine learning and AI do critically depend on at least three key components: (i) the risk function, which is the expectation of the loss function, (ii) the function space, which is often called the hypothesis space, and…
Most of the existing classification methods are aimed at minimization of empirical risk (through some simple point-based error measured with loss function) with added regularization. We propose to approach this problem in a more information…
We revisit the problem of asymmetric binary hypothesis testing against a composite alternative hypothesis. We introduce a general framework to treat such problems when the alternative hypothesis adheres to certain axioms. In this case we…
New upper bounds on the relative entropy are derived as a function of the total variation distance. One bound refines an inequality by Verd\'{u} for general probability measures. A second bound improves the tightness of an inequality by…
All inverse problems rely on data to recover unknown parameters, yet not all data are equally informative. This raises the central question of data selection. A distinctive challenge in PDE-based inverse problems is their inherently…
Information-theoretic measures have been widely adopted in the design of features for learning and decision problems. Inspired by this, we look at the relationship between i) a weak form of information loss in the Shannon sense and ii) the…
We present a general approach, based on exponential inequalities, to derive bounds on the generalization error of randomized learning algorithms. Using this approach, we provide bounds on the average generalization error as well as bounds…
Bounds on information combining are entropic inequalities that determine how the information, or entropy, of a set of random variables can change when they are combined in certain prescribed ways. Such bounds play an important role in…
The mutual information between a set of stimuli and the elicited neural responses is compared to the corresponding decoded information. The decoding procedure is presented as an artificial distortion of the joint probabilities between…
Recently, it has been claimed that a linear relationship between a measure of information content and word length is expected from word length optimization and it has been shown that this linearity is supported by a strong correlation…
Relativistic quantum information combines the informational approach to understanding and using quantum mechanics systems - quantum information - with the relativistic view of the universe. In this introductory review we examine key results…
A simple method is shown to provide optimal variational bounds on $f$-divergences with possible constraints on relative information extremums. Known results are refined or proved to be optimal as particular cases.
The variance of (relative) surprisal, also known as varentropy, so far mostly plays a role in information theory as quantifying the leading order corrections to asymptotic i.i.d.~limits. Here, we comprehensively study the use of it to…
The empirical loss, commonly referred to as the average loss, is extensively utilized for training machine learning models. However, in order to address the diverse performance requirements of machine learning models, the use of the…