Related papers: Generalized Landauer bound from absolute irreversi…
One of the fundamental challenges in the deep learning community is to theoretically understand how well a deep neural network generalizes to unseen data. However, current approaches often yield generalization bounds that are either too…
New upper bounds on the relative entropy are derived as a function of the total variation distance. One bound refines an inequality by Verd\'{u} for general probability measures. A second bound improves the tightness of an inequality by…
Discrete state spaces represent a major computational challenge to statistical inference, since the computation of normalisation constants requires summation over large or possibly infinite sets, which can be impractical. This paper…
Algorithm- and data-dependent generalization bounds are required to explain the generalization behavior of modern machine learning algorithms. In this context, there exists information theoretic generalization bounds that involve (various…
Landauer's principle makes a strong connection between information theory and thermodynamics by stating that erasing a one-bit memory at temperature $T_0$ requires an average energy larger than $W_{LB}=k_BT_0 \ln2$, with $k_B$ Boltzmann's…
Landauer's principle states that the erasure of one bit of information requires the free energy kT ln 2. We argue that the reliability of the bit erasure process is bounded by the accuracy inherent in the statistical state of the energy…
In this paper, we present the Bennett-type generalization bounds of the learning process for i.i.d. samples, and then show that the generalization bounds have a faster rate of convergence than the traditional results. In particular, we…
It is widely accepted that information erasure entails heat dissipation. Here we analyze asymmetric memory states to show that this energy cost can be shuffled around to any step in a write-erase cycle and need not accompany the logically…
Landauer's Principle states that the energy cost of information processing must exceed the product of the temperature and the change in Shannon entropy of the information-bearing degrees of freedom. However, this lower bound is achievable…
Computations implemented on a physical system are fundamentally limited by the laws of physics. A prominent example for a physical law that bounds computations is the Landauer principle. According to this principle, erasing a bit of…
Landauer's principle states that the erasure of information generates a corresponding amount of entropy in the environment. We show that Landauer's principle provides an intuitive basis for Holevo bound on the classical capacity of a…
We study inductive matrix completion (matrix completion with side information) under an i.i.d. subgaussian noise assumption at a low noise regime, with uniform sampling of the entries. We obtain for the first time generalization bounds with…
We present a general framework to study uniqueness, stability and reconstruction for infinite-dimensional inverse problems when only a finite-dimensional approximation of the measurements is available. For a large class of inverse problems…
Landauer's principle provides a perspective on the physical meaning of information as well as on the minimum working cost of information processing. Whereas most studies have related the decrease in entropy during a computationally…
We study the thermodynamics of a Brownian particle under the influence of a time multiplexed harmonic potential of finite width. The memory storage mechanism and the erasure protocol realized by time multiplexed potentials are utilized to…
In statistical learning theory, generalization error is used to quantify the degree to which a supervised machine learning algorithm may overfit to training data. Recent work [Xu and Raginsky (2017)] has established a bound on the…
Information erasure inevitably leads to heat dissipation. Minimizing this dissipation will be crucial for developing small-scale information processing systems, but little is known about the optimal procedures required. We have obtained…
In thermodynamics one considers thermal systems and the maximization of entropy subject to the conservation of energy. A consequence is Landauer's erasure principle, which states that the erasure of 1 bit of information requires a minimum…
We present new excess risk bounds for general unbounded loss functions including log loss and squared loss, where the distribution of the losses may be heavy-tailed. The bounds hold for general estimators, but they are optimized when…
In this paper, we study the problem of signal estimation from noisy non-linear measurements when the unknown $n$-dimensional signal is in the range of an $L$-Lipschitz continuous generative model with bounded $k$-dimensional inputs. We make…