Related papers: Physics in Next-token Prediction
We focus attention on some particular thermodynamic relations (PTR). Using information theory concepts we show that, for a reversible process, microscopic considerations related to these PTR make the concomitant informational contents of…
We argue that in-context learning (ICL) predictably arises from standard self-supervised next-token pretraining, rather than being an exotic emergent property. This work establishes the foundational principles of this emergence by focusing…
We review the physical foundations of Landauer's Principle, which relates the loss of information from a computational process to an increase in thermodynamic entropy. Despite the long history of the Principle, its fundamental rationale and…
The Conservation of Energy plays a pivotal part in the development of the physical sciences. With the growth of computation and the study of other discrete token based systems such as the genome, it is useful to ask if there are…
Starting from Landauer's slogan "information is physical," we revise and modify Landauer's principle stating that the erasure of information has a minimal price in the form of a certain quantity of free energy. We establish a direct link…
The solution of time dependent differential equations with neural networks has attracted a lot of attention recently. The central idea is to learn the laws that govern the evolution of the solution from data, which might be polluted with…
The problem of formulating thermodynamics in a relativistic scenario remains unresolved, although many proposals exist in the literature. The challenge arises due to the intrinsic dynamic structure of spacetime as established by the general…
Traditional form of the second law of thermodynamics is strongly restricted by three conditions: One is the initial joint state of the system and surroundings should be a product state, so that there exists no initial correlations. The…
In a previous paper [1] we considered the question, "What underlying property of nature is responsible for the second law?" A simple answer can be stated in terms of information: The fundamental loss of information gives rise to the second…
We provide a derivation of quantum theory in which the existence of an energy observable that generates the reversible dynamics follows directly from information-theoretic principles. Our first principle is that every reversible dynamics is…
Thermodynamics and information have intricate interrelations. Often thermodynamics is considered to be the logical premise to justify that information is physical - through Landauer's principle -, thereby also linking information and…
We consider the statistical irreversibility and its compatibility with the reversible dynamics. The role played by the observation is analyzed in detail. It makes our previous proof for the second law of thermodynamics clearer. On this…
The first law of thermodynamics, which governs energy conservation, is traditionally formulated as an equality. Surprisingly, we demonstrate that the first law alone implies a universal Landauer-like inequality linking changes in system…
We speculate whether the second law of thermodynamics has more to do with Turing machines than steam pipes. It states the logical reversibility of reality as a computation, i.e., the fact that no information is forgotten: nature computes…
Predicting future observations plays a central role in machine learning, biology, economics, and many other fields. It lies at the heart of organizational principles such as the variational free energy principle and has even been shown --…
We initiate an investigation into the optimization properties of next-token prediction (NTP), the dominant training paradigm for modern language models. Specifically, we study the structural properties of the solutions selected by…
Large language models (LLMs) have demonstrated remarkable in-context learning (ICL) abilities. However, existing theoretical analysis of ICL primarily exhibits two limitations: (a) Limited i.i.d. Setting. Most studies focus on supervised…
The rapid scaling of artificial intelligence models has revealed a fundamental tension between model capacity (storage) and inference efficiency (computation). While classical information theory focuses on transmission and storage limits,…
Connections between information theory and thermodynamics have proven to be very useful to establish bounding limits for physical processes. Ideas such as Landauer's erasure principle and information assisted work extraction have greatly…
Second law of thermodynamics can be apparently violated for systems whose dynamics depends on acquired information by measurement. However, when one consider measurement and erasure process together along with the system it saves the second…