Related papers: Learning rate matrix and information-thermodynamic…

The learning rate is an information-theoretical quantity for bipartite Markov chains describing two coupled subsystems. It is defined as the rate at which transitions in the downstream subsystem tend to increase the mutual information…

This paper employs a formal connection of machine learning with thermodynamics to characterize the quality of learnt representations for transfer learning. We discuss how information-theoretic functional such as rate, distortion and…

Lateral predictive coding is a recurrent neural network which creates energy-efficient internal representations by exploiting statistical regularity in sensory inputs. Here we investigate the trade-off between information robustness and…

Information thermodynamics relates the rate of change of mutual information between two interacting subsystems to their thermodynamics when the joined system is described by a bipartite stochastic dynamics satisfying local detailed balance.…

In the recent progress in nonequilibrium thermodynamics, information has been recognized as a kind of thermodynamic resource that can drive thermodynamic current without any direct energy injection. In this paper, we establish the framework…

Fluctuations in biochemical networks, e.g., in a living cell, have a complex origin that precludes a description of such systems in terms of bipartite or multipartite processes, as is usually done in the framework of stochastic and/or…

For sensory networks, we determine the rate with which they acquire information about the changing external conditions. Comparing this rate with the thermodynamic entropy production that quantifies the cost of maintaining the network, we…

Connections between information theory and thermodynamics have proven to be very useful to establish bounding limits for physical processes. Ideas such as Landauer's erasure principle and information assisted work extraction have greatly…

We explore the dynamics of information systems. We show that the driving force for information dynamics is determined by both the information landscape and information flux which determines the equilibrium time reversible and the…

A unified thermodynamic formalism describing the efficiency of learning is proposed. First, we derive an inequality, which is more strength than Clausius's inequality, revealing the lower bound of the entropy-production rate of a subsystem.…

This paper demonstrates dynamic hyper-parameter setting, for deep neural network training, using Mutual Information (MI). The specific hyper-parameter studied in this paper is the learning rate. MI between the output layer and true outcomes…

Agents that learn to select optimal actions represent a prominent focus of the sequential decision-making literature. In the face of a complex environment or constraints on time and resources, however, aiming to synthesize such an optimal…

Thermodynamic tradeoff relations quantify the fundamental concept of ``no free lunch'' in the physical world, suggesting that faster and more precise physical processes come at a higher thermodynamic cost. The key elements in these tradeoff…

Virtually every organism gathers information about its noisy environment and builds models from that data, mostly using neural networks. Here, we use stochastic thermodynamics to analyse the learning of a classification rule by a neural…

We discuss a relationship between information geometry and the Glansdorff-Prigogine criterion for stability. For the linear master equation, we found a relation between the line element and the excess entropy production rate. This relation…

An unsupervised learning procedure based on maximizing the mutual information between the outputs of two networks receiving different but statistically dependent inputs is analyzed (Becker and Hinton, Nature, 355, 92, 161). For a generic…

This paper proposes an information-theoretic representation learning framework, named conditional information flow maximization, to extract noise-invariant sufficient representations for the input data and target task. It promotes the…

Reinforcement Learning (RL) models have continually evolved to navigate the exploration - exploitation trade-off in uncertain Markov Decision Processes (MDPs). In this study, I leverage the principles of stochastic thermodynamics and system…

Irreversible processes accomplished in a fixed time involve nonlinearly coupled flows of matter, energy, and information. Here, using entropy production as an example, we show how thermodynamic uncertainty relations and speed limits on…

Biological organisms are open, adaptve systems that can respond to changes in environment in specific ways. Adaptation and response can be posed as an optimization problem, with a tradeoff between the benefit obtained from a response and…