Related papers: Physics in Next-token Prediction
Dynamical systems describe how a physical system evolves over time. Physical processes can evolve faster or slower in different environmental conditions. We use time-warping as rescaling the time in a model of a physical system. This thesis…
We study the informational underpinnings of thermodynamics and statistical mechanics, using an abstract framework, general probabilistic theories, capable of describing arbitrary physical theories. This allows one to abstract the…
Can general-purpose AI architectures go beyond prediction to discover the physical laws governing the universe? True intelligence relies on "world models" -- causal abstractions that allow an agent to not only predict future states but…
Thermodynamics is based on the notions of energy and entropy. While energy is the elementary quantity governing physical dynamics, entropy is the fundamental concept in information theory. In this work, starting from first principles, we…
The standard relations between information theory and thermodynamics are challenged. The Szilard engine is revisited and the bound proposed by Landauer is replaced by a different one which includes errors in information processing. Instead…
Zellner (1988) modeled statistical inference in terms of information processing and postulated the Information Conservation Principle (ICP) between the input and output of the information processing block, showing that this yielded Bayesian…
Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM…
Recently, there has been a considerable progress on the issue of the thermodynamic second law, which is known as the law of entropy increase or irreversibility. In particular, a novel symmetry known as the Gallavotti-Cohen symmetry is found…
This article is intended for physical scientists who wish to gain deeper insights into machine learning algorithms which we present via the domain they know best, physics. We begin with a review of two energy-based machine learning…
Continual Text Classification (CTC) aims to continuously classify new text data over time while minimizing catastrophic forgetting of previously acquired knowledge. However, existing methods often focus on task-specific knowledge,…
Information causality (IC) was one of the first principles that have been invoked to bound the set of quantum correlations. For some families of correlations, this principle recovers exactly the boundary of the quantum set; for others,…
Recently observed empirical scaling laws describe the performance of foundation-type models as three independent key quantities -- dataset size, compute, and model parameters -- are modified. Extracting these scaling laws informs the…
Feed-forward deep neural networks have been used extensively in various machine learning applications. Developing a precise understanding of the underling behavior of neural networks is crucial for their efficient deployment. In this paper,…
Strictly speaking, Newton's second law of motion is only an approximation of the so-called relativistic dynamics, i.e., Einstein's modification of the second law based on his theory of special relativity. Although the approximation is…
Landauer's Principle that information loss from a computation implies entropy increase can be rigorously proved from mathematical physics. However, carefully examining its detailed formulation reveals that the traditional identification of…
Understanding the connections between information and thermodynamics has been among the most visible applications of stochastic thermodynamics. While recent theoretical advances have established that the second law of thermodynamics sets…
One of the outstanding challenges to information processing is the eloquent suppression of energy consumption in execution of logic operations. Landauer principle sets an energy constraint in deletion of a classical bit of information.…
We address a new setting where the second law is under question: thermalizations in a quantum superposition of causal orders, enacted by the so-called quantum switch. This superposition has been shown to be associated with an increase in…
This paper introduces time into information theory, gives a more accurate definition of information, and unifies the information in cognition and Shannon information theory. Specially, we consider time as a measure of information, giving a…
We introduce the \emph{Arrow Language Model}, a neural architecture derived from an intuitionistic-logic interpretation of next-token prediction. Instead of representing tokens as additive embeddings mixed by attention, we encode a prefix…