Related papers: Uniform Scaling Limits in AdamW-Trained Transforme…
Local M-smoothers are interesting and important signal and image processing techniques with many connections to other methods. In our paper we derive a family of partial differential equations (PDEs) that result in one, two, and three…
Transformer models have achieved state-of-the-art results across a diverse range of domains. However, concern over the cost of training the attention mechanism to learn complex dependencies between distant inputs continues to grow. In…
This manuscript investigates the one-pass stochastic gradient descent (SGD) dynamics of a two-layer neural network trained on Gaussian data and labels generated by a similar, though not necessarily identical, target function. We rigorously…
Anderson localization of particles -- the complete halt of wave transport through multiple scattering and phase coherence -- is a paradigmatic manifestation of quantum interference in disordered media. In three dimensions, the scaling…
We study the problem of learning Transformer-based sequence models with black-box access to their outputs. In this setting, a learner may adaptively query the oracle with any sequence of vectors and observe the output of the target…
Self-attention in Transformers relies on globally normalized softmax weights, causing all tokens to compete for influence at every layer. When composed across depth, this interaction pattern induces strong synchronization dynamics that…
Entity tracking requires maintaining and updating latent states for entities and attributes over long sequences. Recent task-specific attention operators can compress deep Transformer stacks into a few layers by performing multi-hop state…
This paper addresses the local stabilization problem for semilinear single-track vehicle models with distributed tire friction dynamics, represented as interconnections of ordinary differential equations (ODEs) and hyperbolic partial…
Building scalable models to learn from diverse, multimodal data remains an open challenge. For vision-language data, the dominant approaches are based on contrastive learning objectives that train a separate encoder for each modality. While…
We present a theoretical analysis of the training process for a single-layer GAN fed by high-dimensional input data. The training dynamics of the proposed model at both microscopic and macroscopic scales can be exactly analyzed in the…
This work addresses two major issues of end-to-end learned image compression (LIC) based on deep neural networks: variable-rate learning where separate networks are required to generate compressed images with varying qualities, and the…
Sequence models face a fundamental tradeoff between memory capacity and computational efficiency. Transformers achieve expressive context modeling at quadratic cost, while linear attention and state-space models run in linear time by…
Most ordinary differential equation (ODE) models used to describe biological or physical systems must be solved approximately using numerical methods. Perniciously, even those solvers which seem sufficiently accurate for the forward…
Transformers have transformed modern machine learning, driving breakthroughs in computer vision, natural language processing, and robotics. At the core of their success lies the attention mechanism, which enables the modeling of global…
We use a stochastic series expansion quantum Monte Carlo method to study the phase diagram of the one-dimensional extended Hubbard model at half filling for small to intermediate values of the on-site (U) and nearest-neighbor (V)…
Solids undergoing a transition from order to disorder experience the proliferation of topological defects. The melting process generates transient quantum states. However, their dynamical nature with femtosecond lifetime hinders exploration…
We consider the Kob-Andersen model, a cooperative lattice gas with kinetic constraints which has been widely analyzed in the physics literature in connection with the study of the liquid/glass transition. We consider the model in a finite…
Continuous-time neural processes are performant sequential decision-makers that are built by differential equations (DE). However, their expressive power when they are deployed on computers is bottlenecked by numerical DE solvers. This…
As power systems transition toward renewable-rich and inverter-dominated operations, accurate time-domain dynamic analysis becomes increasingly critical. Such analysis supports key operational tasks, including transient stability…
Understanding the training dynamics of transformers is important to explain the impressive capabilities behind large language models. In this work, we study the dynamics of training a shallow transformer on a task of recognizing…