Related papers: Physics in Next-token Prediction
The averaged steady-state surprisal links a driven stochastic system's information processing to its nonequilibrium thermodynamic response. By explicitly accounting for the effects of nonequilibrium steady states, a decomposition of the…
This study aims to quantify and visualize the degradation of fidelity (information degradation) that inevitably accompanies the replication of information within the framework of information thermodynamics and to propose an…
We propose one of the very few constructive consequences of the second law of thermodynamics. More specifically, we present protocols for secret-key establishment and multiparty computation the security of which is based fundamentally on…
I argue here that both the first and second laws of thermodynamics, generally understood to be quintessentially physical in nature, can be equally well described as being about certain types of information without the need to invoke…
Quantum mechanics and information theory are among the most important scientific discoveries of the last century. Although these two areas initially developed separately it has emerged that they are in fact intimately related. In this…
Using a mechanical cantilever submitted to electrostatic feedback control, we investigate the thermodynamic properties of an information engine that extracts work from thermal fluctuations. The cantilever position is rapidly sampled and the…
The foundations of Statistical Mechanics can be recovered almost in their entirety from the Principle of Maximum Entropy. In this work we show that its non-equilibrium generalization, the Principle of Maximum Caliber (Jaynes, 1980), when…
From the principle of maximum entropy for a closed system in thermal equilibrium, for the first instance a clear relation is shown to exist between total entropy S (in terms of arrangements of particles) and the classical expression for the…
Neural scaling laws define a predictable relationship between a model's parameter count and its performance after training in the form of a power law. However, most research to date has not explicitly investigated whether scaling laws can…
A central result that arose in applying information theory to the stochastic thermodynamics of nonlinear dynamical systems is the Information-Processing Second Law (IPSL): the physical entropy of the universe can decrease if compensated by…
We provide a simple physical interpretation, in the context of the second law of thermodynamics, to the information inequality (a.k.a. the Gibbs' inequality, which is also equivalent to the log-sum inequality), asserting that the relative…
Next-frame prediction is a useful and powerful method for modelling and understanding the dynamics of video data. Inspired by the empirical success of causal language modelling and next-token prediction in language modelling, we explore the…
Machine learning is playing an increasing role in the physical sciences and significant progress has been made towards embedding domain knowledge into models. Less explored is its use to discover interpretable physical laws from data. We…
We develop a method to learn physical systems from data that employs feedforward neural networks and whose predictions comply with the first and second principles of thermodynamics. The method employs a minimum amount of data by enforcing…
Given a sequence of tokens, such as words, the task of next-token prediction is to predict the next-token conditional probability distribution. Decoder-only transformers have become effective models for this task, but their properties are…
The association of information with entropy has been argued on plausibility arguments involving the operation of imaginary engines and beings, and it is not a universal theorem. In this paper, a theorem by Charles Bennett on reversible…
Scaling laws in deep learning -- empirical power-law relationships linking model performance to resource growth -- have emerged as simple yet striking regularities across architectures, datasets, and tasks. These laws are particularly…
Standard next-token prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse one-hot supervision leaves the latent representation space under-constrained, allowing…
The development of stochastic thermodynamics during the last decades prompted the discovery of novel nonequilibrium relations refining our understanding of the second law in small fluctuating systems and its connection with information…
The Kullback-Leibler inequality is a way of comparing any two density matrices. A technique to set up the density matrix for a physical system is to use the maximum entropy principle, given the entropy as a functional of the density matrix,…