Related papers: Stochastic Thermodynamics of Learning Parametric P…
We define {\em predictive information} $I_{\rm pred} (T)$ as the mutual information between the past and the future of a time series. Three qualitatively different behaviors are found in the limit of large observation times $T$: $I_{\rm…
A general formulation of stochastic thermodynamics is presented for open systems exchanging energy and particles with multiple reservoirs. By introducing a partition in terms of "macrostates" (e.g. sets of "microstates"), the consequence on…
A key tenet of the Transactional Interpretation of Quantum Mechanics is the idea that photon absorption localizes the absorbing material system. In doing so, it measures the location of the absorber and hence reduces information entropy,…
We have presented first an axiomatic derivation of Boltzmann entropy on the basis of two axioms consistent with two basic properties of thermodynamic entropy. We have then studied the relationship between Boltzmann entropy and information…
In a previous paper, we introduced an axiomatic system for information thermodynamics, deriving an entropy function that includes both thermodynamic and information components. From this function we derived an entropic probability…
Learning is a fundamental characteristic of living systems, enabling them to comprehend their environments and make informed decisions. These decision-making processes are inherently influenced by available information about their…
In these decades, it has been revealed that there is rich information-theoretic structure in thermodynamics of out-of-equilibrium systems in both the classical and quantum regimes. This has led to the fruitful interplay among statistical…
We show that the mutual information between the representation of a learning machine and the hidden features that it extracts from data is bounded from below by the relevance, which is the entropy of the model's energy distribution. Models…
A Boltzmann machine is a stochastic neural network that has been extensively used in the layers of deep architectures for modern machine learning applications. In this paper, we develop a Boltzmann machine that is capable of modelling…
Predicting the physico-chemical properties of pure substances and mixtures is a central task in thermodynamics. Established prediction methods range from fully physics-based ab-initio calculations, which are only feasible for very simple…
Thermodynamics establishes that information acquired through measurement can be converted into work, as exemplified by Maxwell's demon and Szilard engines. Most experimental realizations of information engines, however, implicitly assume…
Markedly increased computational power and data acquisition have led to growing interest in data-driven inverse dynamics problems. These seek to answer a fundamental question: What can we learn from time series measurements of a complex…
There are three ways to conceptualize entropy: entropy as an extensive thermodynamic quantity of physical systems (Clausius, Boltzmann, Gibbs), entropy as a measure for information production of ergodic sources (Shannon), and entropy as a…
Perfectly rational decision-makers maximize expected utility, but crucially ignore the resource costs incurred when determining optimal actions. Here we propose an information-theoretic formalization of bounded rational decision-making…
Beyond neural scaling laws, little is known about the laws underlying large language models (LLMs). We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fresh insights into LLM training dynamics. On the theoretical…
Thermoelectric materials provide a sustainable way to convert waste heat into electricity. However, data-driven discovery and optimization of these materials are challenging because of a lack of a reliable database. Here we developed a…
Machine learning, with its remarkable ability for retrieving information and identifying patterns from data, has emerged as a powerful tool for discovering governing equations. It has been increasingly informed by physics, and more recently…
The rapid scaling of artificial intelligence models has revealed a fundamental tension between model capacity (storage) and inference efficiency (computation). While classical information theory focuses on transmission and storage limits,…
Information is central to thermodynamics, providing the grounds to the formulation of the theory in powerful abstract statistical terms. One must not forget, however, that, as put by Landauer, {\it information is physical}. This means that…
We consider reinforcement learning in parameterized Markov Decision Processes (MDPs), where the parameterization may induce correlation across transition probabilities or rewards. Consequently, observing a particular state transition might…