Related papers: Learning rate matrix and information-thermodynamic…
We show that a rate of conditional Shannon entropy reduction, characterizing the learning of an internal process about an external process, is bounded by the thermodynamic entropy production. This approach allows for the definition of an…
We present novel results that relate energy and information transfer with sensitivity to initial conditions in chaotic multi-dimensional Hamiltonian systems. We show the relation among Kolmogorov-Sinai entropy, Lyapunov exponents, and upper…
We generalize stochastic thermodynamics to include information reservoirs. Such information reservoirs, which can be modeled as a sequence of bits, modify the second law. For example, work extraction from a system in contact with a single…
Information dynamics is an emerging description of information processing in complex systems which describes systems in terms of intrinsic computation, identifying computational primitives of information storage and transfer. In this paper…
Multi-modal contrastive learning as a self-supervised representation learning technique has achieved great success in foundation model training, such as CLIP~\citep{radford2021learning}. In this paper, we study the theoretical properties of…
Learning with hidden variables is a central challenge in probabilistic graphical models that has important implications for many real-life problems. The classical approach is using the Expectation Maximization (EM) algorithm. This…
We relate the information exchange between two stochastic systems to the nonequilibrium entropy production in the whole system. By deriving a general formula that decomposes the total entropy production into the thermodynamic and…
This letter exposes a tight connection between the thermodynamic efficiency of information processing and predictive inference. A generalized lower bound on dissipation is derived for partially observable information engines which are…
Understanding the pattern formation in communities has been at the center of attention in various fields. Here we introduce a novel model, called an "information-particle model," which is based on the reaction-diffusion model and the…
On the one hand, the dissipated heat of a thermodynamic work extraction process upper bounds the non-predictive information, which the associated system encodes about its environment. Thus, emergent information processing capabilities can…
The Landauer's principle, a cornerstone of information thermodynamics, provides a fundamental lower bound on the energetic cost of information erasure in terms of the information content change. However, its traditional formulation is…
Analysing how information flows along the layers of a multilayer perceptron is a topic of paramount importance in the field of artificial neural networks. After framing the problem from the point of view of information theory, in this…
Identifying the origin of nonequilibrium characteristics in a generic interacting system having multiple degrees of freedom is a challenging task. In this context, information theoretic measures such as mutual information and related…
A quantum thermal machine is an open quantum system that enables the conversion between heat and work at the micro or nano-scale. Optimally controlling such out-of-equilibrium systems is a crucial yet challenging task with applications to…
By searching for shared inductive biases across tasks, meta-learning promises to accelerate learning on novel tasks, but with the cost of solving a complex bilevel optimization problem. We introduce and rigorously define the trade-off…
We demonstrate that a wide array of machine learning algorithms are specific instances of one single paradigm: reciprocal learning. These instances range from active learning over multi-armed bandits to self-training. We show that all these…
We derive the amount of information retrieved by a quantum measurement in estimating an unknown maximally entangled state, along with the pertaining disturbance on the state itself. The optimal tradeoff between information and disturbance…
The established thermodynamic formalism of chaotic dynamics, valid at statistical equilibrium, is here generalized to systems out of equilibrium, that have yet to relax to a steady state. A relation between information, escape rate, and the…
Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss. This puzzling contradiction necessitates new approaches to the study of overfitting. Here we quantify…
We consider long-lived agents who interact repeatedly in a social network. In each period, each agent learns about an unknown state by observing a private signal and her neighbors' actions from the previous period before choosing her own…