Related papers: Thermodynamic Irreversibility of Training Algorith…
Irreversibility is a fundamental concept with important implications at many levels. It pinpoints the fundamental difference between the intrinsically reversible microscopic equations of motion and the unidirectional arrow of time that…
There is a relation between the irreversibility of thermodynamic processes as expressed by the breaking of time-reversal symmetry, and the entropy production in such processes. We explain on an elementary mathematical level the relations…
The emergence of irreversibility in physical processes, despite the fundamentally reversible nature of quantum mechanics, remains an open question in physics. This thesis explores the intricate relationship between quantum mechanics and…
Visibility algorithms are a family of methods to map time series into networks, with the aim of describing the structure of time series and their underlying dynamical properties in graph-theoretical terms. Here we explore some properties of…
Deep learning systems achieve remarkable empirical performance, yet the stability of the training process itself remains poorly understood. Training unfolds as a high-dimensional dynamical system in which small perturbations to…
Learning performed over finite time is inherently irreversible. In Part~I of this series, we modeled learning as a transport process in the space of parameter distributions and derived the Epistemic Speed Limit (ESL), which lower-bounds…
In stochastic thermodynamics, the entropy production of a thermodynamic system is defined by the irreversibility measured by the logarithm of the ratio of the path probabilities in the forward and reverse processes. We derive the relation…
Developing a thermodynamic theory of computation is a challenging task at the interface of non-equilibrium thermodynamics and computer science. In particular, this task requires dealing with difficulties such as stochastic halting times,…
Thermodynamics has a clear arrow of time, characterized by the irreversible approach to equilibrium. This stands in contrast to the laws of microscopic theories, which are invariant under time-reversal. Foundational discussions of this…
This work introduces a novel, simple, and flexible method to quantify irreversibility in generic high-dimensional time series based on the well-known mapping to a binary classification problem. Our approach utilizes gradient boosting for…
The dynamics of irreversible relaxation of non-equilibrium macroscopic systems is discussed. Arguments are developed showing that the general process is supported by two independent successive mechanisms. One is mixing and it follows pure…
The dynamics of learning in modern large AI systems is hierarchical, often characterized by abrupt, qualitative shifts akin to phase transitions observed in physical systems. While these phenomena hold promise for uncovering the mechanisms…
Reproducibility is an increasing concern in Artificial Intelligence (AI), particularly in the area of Deep Learning (DL). Being able to reproduce DL models is crucial for AI-based systems, as it is closely tied to various tasks like…
Algorithmic entropy can be seen as a special case of entropy as studied in statistical mechanics. This viewpoint allows us to apply many techniques developed for use in thermodynamics to the subject of algorithmic information theory. In…
Learning systems acquire structured internal representations from data, yet classical information-theoretic results state that deterministic transformations do not increase information. This raises a fundamental question: how can learning…
We examine stochastic processes that are used to model nonequilibrium processes (e.g, pulling RNA or dragging colloids) and so deliberately violate detailed balance. We argue that by combining an information-theoretic measure of…
We propose a new approach concerning the introduction of time-irreversibility in statistical mechanics. It is based on a transition function defined in terms of path integral and verifying a time-irreversible equation. We show first how…
Time-reversal symmetry arises naturally as a structural property in many dynamical systems of interest. While the importance of hard-wiring symmetry is increasingly recognized in machine learning, to date this has eluded time-reversibility.…
We demonstrate that irreversibility arises from the principle of microscopic reversibility and the presence of memory in the time evolution of a single copy of a system driven by a protocol. We introduce microscopic reversibility by using…
Even though irreversibility is one of the major hallmarks of any real life process, an actual understanding of irreversible processes remains still mostly semiempirical. In this paper we formulate a thermodynamic uncertainty principle for…