Related papers: Physics in Next-token Prediction
The characterisation of information processing is an important task in complex systems science. Information dynamics is a quantitative methodology for modelling the intrinsic information processing conducted by a process represented as a…
Identifying the dynamics of physical systems requires a machine learning model that can assimilate observational data, but also incorporate the laws of physics. Neural Networks based on physical principles such as the Hamiltonian or…
Entanglement is central both to the foundations of quantum theory and, as a novel resource, to quantum information science. The theory of entanglement establishes basic laws, such as the non-increase of entanglement under local operations,…
We introduce a methodology for seeking conservation laws within a Hamiltonian dynamical system, which we term ``neural deflation''. Inspired by deflation methods for steady states of dynamical systems, we propose to {iteratively} train a…
We develop a thermodynamic theory of algorithmic catalysis within the watts-per-intelligence framework, identifying reusable computational structures that reduce irreversible operations for a task class while satisfying bounded restoration…
Transformer-based language models are trained on large datasets to predict the next token given an input sequence. Despite this simple training objective, they have led to revolutionary advances in natural language processing. Underlying…
The link between information and energy introduces the observer and their knowledge into the understanding of a fundamental quantity of physics. Two approaches compete to account for this link, Brillouin's negentropy law of information and…
In this letter, motivated by the question that whether the empirical fitting of data by neural network can yield the same structure of physical laws, we apply the neural network to a simple quantum mechanical two-body scattering problem…
A general information-theoretic framework for deriving physical laws is presented and a principle of informational physics is enunciated within its context. Existing approaches intended to derive physical laws from information-theoretic…
While current AI-driven methods excel at deriving empirical models from individual experiments, a significant challenge remains in uncovering the common fundamental physics that underlie these models -- a task at which human physicists are…
Understanding the geometric properties of gradient descent dynamics is a key ingredient in deciphering the recent success of very large machine learning models. A striking observation is that trained over-parameterized models retain some…
We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, thereby forming a more challenging pretraining objective. Our…
Using a recent published formula for the number of information N that results from Landauer's principle we obtain an expression for the cosmological constant Lambda. Next, assuming the universe as a system of mass M satisfying Landauer's…
In this paper I apply newly-proposed information-theoretic principles to thermodynamic work extraction. I show that if it is possible to extract work deterministically from a physical system prepared in any one of a set of states, then…
Learning diverse and high-fidelity traffic simulations from human driving demonstrations is crucial for autonomous driving evaluation. The recent next-token prediction (NTP) paradigm, widely adopted in large language models (LLMs), has been…
This work consists in the theorical development on the analysis of the Thermodynamic Laws and thermodynamic systems in relative motion, according to the laws of Classical Mechanics. The difference of this work for many of the literature is…
We review and investigate the general theory of thermodynamics of computation, and derive the fundamental inequalities that set the lower bounds of the work requirement and the heat emission during a computation. These inequalities…
We describe a simple framework for teaching the principles that underlie the dynamical laws of transport: Fick's law of diffusion, Fourier's law of heat flow, the Newtonian viscosity law, and mass-action laws of chemical kinetics. In…
Many complex physical systems admit natural decomposition into an exactly solvable component and a perturbative correction. Rather than training neural networks to learn complete trajectories from scratch, we introduce Neural Network…
We consider the generalized second law of black hole thermodynamics in the light of quantum information theory, in particular information erasure and Landauer's principle (namely, that erasure of information produces at least the equivalent…