Related papers: How Do Transformers "Do" Physics? Investigating th…
Transformers have achieved state-of-the-art performance in numerous tasks. In this paper, we propose a continuous-time formulation of transformers. Specifically, we consider a dynamical system whose governing equation is parametrized by…
White noise analysis is used to derive the propagator of an open quantum system consisting of a harmonic oscillator which is coupled to an environment consisting of N multimode harmonic oscillators. The quantum propagators are obtained…
A similarity transformation is constructed through which a system of particles interacting with inverse-square two-body and harmonic potentials in one dimension, can be mapped identically, to a set of free harmonic oscillators. This…
Transformer networks excel in scientific applications. We explore two scenarios in ultra-high-energy cosmic ray simulations to examine what these network architectures learn. First, we investigate the trained positional encodings in air…
For many-particle systems defined on lattices we investigate the global structure of effective Hamiltonians and observables obtained by means of a suitable basis transformation. We study transformations which lead to effective Hamiltonians…
Transformers have demonstrated exceptional in-context learning capabilities, yet the theoretical understanding of the underlying mechanisms remains limited. A recent work (Elhage et al., 2021) identified a ``rich'' in-context mechanism…
The quantum Otto cycle serves as a bridge between the macroscopic world of heat engines and the quantum regime of thermal devices composed from a single element. We compile recent studies of the quantum Otto cycle with a harmonic oscillator…
The attention mechanism in its standard implementation contains extraneous rotational degrees of freedom that are carried through computation but do not affect model activations or outputs. We introduce a simple symmetry-breaking protocol…
We observe a large number of functions differing from each other only by a translation parameter. While the main pattern is unknown, we propose to estimate the shift parameters using $M$-estimators. Fourier transform enables to transform…
The paper introduces a simple quantum model to calculate in a general way allowed frequencies and energy levels of the anharmonic oscillator. The theoretical basis of the approach has been introduced in two early papers aimed to infer the…
Short text classification is a crucial and challenging aspect of Natural Language Processing. For this reason, there are numerous highly specialized short text classifiers. However, in recent short text research, State of the Art (SOTA)…
Synchronous generators and inverter-based resources are complex systems with dynamics that cut across multiple intertwined physical domains and control loops. Modeling individual generators and inverters is, in itself, a very involved…
The Legendre transform expresses dynamics of a classical system through first-order Hamiltonian equations. We consider coherent state transforms with a similar effect in quantum mechanics: they reduce certain quantum Hamiltonians to…
In particle physics, the fundamental forces are subject to symmetries called gauge invariance. It is a redundancy in the mathematical description of any physical system. In this article I will demonstrate that the transformer architecture…
The three dimensional harmonic oscillator model including a cranking term is used for an energy variational calculation. Energy minima are found under variation of the three oscillator frequencies determining the shape of the system for…
In spite of their huge success, transformer models remain difficult to scale in depth. In this work, we develop a unified signal propagation theory and provide formulae that govern the moments of the forward and backward signal through the…
Foundation models must handle multiple generative processes, yet mechanistic interpretability largely studies capabilities in isolation; it remains unclear how a single transformer organizes multiple, potentially conflicting "world models".…
Recent research has demonstrated that transformers, particularly linear attention models, implicitly execute gradient-descent-like algorithms on data provided in-context during their forward inference step. However, their capability in…
Entanglement represents a pure quantum effect involving two or more particles. Spin systems are good candidates for studying this effect and its relation with other collective phenomena ruled by quantum mechanics. While the presence of…
Normalization and scaling are fundamental preprocessing steps in time series modeling, yet their role in Transformer-based models remains underexplored from a theoretical perspective. In this work, we present the first formal analysis of…