Related papers: Physics in Next-token Prediction
Landauer's principle states that information erasure requires heat dissipation. Landauer's original result focused on equilibrium memories. We here investigate the reset of information stored in a nonequilibrium state of a symmetric…
We use the formalism of 'Maximum Principle of Shannon's Entropy' to derive the general power law distribution function, using what seems to be a reasonable physical assumption, namely, the demand of a constant mean "internal order"…
We investigate a measurement-feedback process of repeated operations with time delay. During a finite-time interval, measurement on the system is performed and the feedback protocol derived from the measurement outcome is applied with time…
Recently, the superior performance of Transformers has made them a more robust and scalable solution for sequence modeling than traditional recurrent neural networks (RNNs). However, the effectiveness of Transformer in capturing long-term…
Many fruitful analogies have emerged between the theories of quantum entanglement and thermodynamics, motivating the pursuit of an axiomatic description of entanglement akin to the laws of thermodynamics. A long-standing open problem has…
Quantum Landauer's principle provides a fundamental lower bound for energy dissipation occurred with information erasure in the quantum regime. While most studies have related the entropy reduction incorporated with the erasure to the lower…
The reversible computation paradigm aims to provide a new foundation for general classical digital computing that is capable of circumventing the thermodynamic limits to the energy efficiency of the conventional, non-reversible digital…
The plethora of Internet of Things (IoT) devices leads to explosive network traffic. The network traffic classification (NTC) is an essential tool to explore behaviours of network flows, and NTC is required for Internet service providers…
We design a heat engine with multi-heat-reservoir, ancillary system and quantum memory. We then derive an inequality related with the second law of thermodynamics, and give a new limitation about the work gain from the engine by analyzing…
We show that the no-deleting and no-cloning principles are implications of information conservation principle. This is unlike in classical physics, where cloning and deleting are possible, independently of information conservation.…
The great power of EQuilibrium (EQ) statistical physics comes from its principled foundations: its First Law (conservation), Second Law (variational tendency principle), and its Legendre Transforms from observables $(U, V, N)$ to their…
In this paper, we prove particle approximations of initial data for systems of conservation laws in two dimensions. This involves approaching the density but also all the additional quantities that could be verified by the model considered.…
We establish a refined version of the Second Law of Thermodynamics for Langevin stochastic processes describing mesoscopic systems driven by conservative or non-conservative forces and interacting with thermal noise. The refinement is based…
Physical phenomena in the real world are often described by energy-based modeling theories, such as Hamiltonian mechanics or the Landau theory, which yield various physical laws. Recent developments in neural networks have enabled the…
Progress in Natural Language Processing (NLP) has been dictated by the rule of more: more data, more computing power and more complexity, best exemplified by the Large Language Models. However, training (or fine-tuning) large dense models…
Energy has no definition, except that given by a conservation principle which essentially amounts to defining it as the elements of an open list of unknown cardinality. Entropy, identified by Shannon as information we lack, has too many…
Cyber and cyber-physical systems equipped with machine learning algorithms such as autonomous cars share environments with humans. In such a setting, it is important to align system (or agent) behaviors with the preferences of one or more…
Next Generation Reservoir Computing (NG-RC) is a modern class of model-free machine learning that enables an accurate forecasting of time series data generated by dynamical systems. We demonstrate that NG-RC can accurately predict full…
In information retrieval (IR) systems, trends and users' interests may change over time, altering either the distribution of requests or contents to be recommended. Since neural ranking approaches heavily depend on the training data, it is…
Deep neural network is the widely applied technology in this decade. In spite of the fruitful applications, the mechanism behind that is still to be elucidated. We study the learning process with a very simple supervised learning encoding…