Related papers: Physics in Next-token Prediction
On the one hand, the dissipated heat of a thermodynamic work extraction process upper bounds the non-predictive information, which the associated system encodes about its environment. Thus, emergent information processing capabilities can…
Even after over 150 years of discussion, the interpretation of the second law of thermodynamics continues to be a source of confusion and controversy in physics. This confusion has been accentuated by recent challenges to the second law and…
A principle of information conservation is shown in abstract terms to rule out probabilistic physical laws, necessitating the existence of state trajectories. It furthermore provides a geometric-thermodynamic mechanism for the appearance of…
In noisy physical reservoirs, the classical information-processing capacity $C_{\mathrm{ip}}$ quantifies how well a linear readout can realize tasks measurable from the input history, yet $C_{\mathrm{ip}}$ can be far smaller than the…
While often treated as abstract algorithmic properties, intelligence and computation are ultimately physical processes constrained by conservation laws. We introduce the Conservation-Congruent Encoding (CCE) framework as a unified,…
We illustrate in terms familiar to modern day science students that: (i) an uncertainty slope mechanism underlies the usefulness of temperature via its reciprocal, which is incidentally around 42 [nats/eV] at the freezing point of water;…
This thesis investigates the connection between quantum theory, thermodynamics and information theory. Theories with structure similar to that of quantum theory are considered, mathematically described by the framework of "Generalized…
Conformal Prediction (CP) is a distribution-free uncertainty estimation framework that constructs prediction sets guaranteed to contain the true answer with a user-specified probability. Intuitively, the size of the prediction set encodes a…
Information entropy and Zipf's law in the field of information theory have been used for studying the disassembly of nuclei in the framework of the isospin dependent lattice gas model and molecular dynamical model. We found that the…
We define {\em predictive information} $I_{\rm pred} (T)$ as the mutual information between the past and the future of a time series. Three qualitatively different behaviors are found in the limit of large observation times $T$: $I_{\rm…
To explain conceptual gap between classical/quantum and other, hypothetical descriptions of world, several principles has been proposed. So far, all these principles have not explicitly included the uncertainty relation. Here we introduce…
It was first suggested by David Z. Albert that the existence of a real, physical non-unitary process (i.e., "collapse") at the quantum level would yield a complete explanation for the Second Law of Thermodynamics (i.e., the increase in…
A unification of thermodynamics and information theory is proposed. It is argued that similarly to the randomness due to collisions in thermal systems, the quenched randomness that exists in data files in informatics systems contributes to…
Even though neural networks enjoy widespread use, they still struggle to learn the basic laws of physics. How might we endow them with better inductive biases? In this paper, we draw inspiration from Hamiltonian mechanics to train models…
On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude. This phenomenon is known as a neural scaling law. Of fundamental importance is…
Next token prediction paradigm has been prevailing for autoregressive models in the era of LLMs. The current default sampling choice for popular LLMs is temperature scaling together with nucleus sampling to balance diversity and coherence.…
Scaling laws aim to accurately predict model performance across different scales. Existing scaling-law studies almost exclusively rely on cross-entropy as the evaluation metric. However, cross-entropy provides only a partial view of…
We investigate the link between information and thermodynamics embodied by Landauer's principle in the open dynamics of a multipartite quantum system. Such irreversible dynamics is described in terms of a collisional model with a finite…
We prove the second law of thermodynamics and the nonequilibirum fluctuation theorem for pure quantum states.The entire system obeys reversible unitary dynamics, where the initial state of the heat bath is not the canonical distribution but…
Neural scaling laws aim to characterize how out-of-sample error behaves as a function of model and training dataset size. Such scaling laws guide allocation of a computational resources between model and data processing to minimize error.…