Related papers: Flowing Through Layers: A Continuous Dynamical Sys…
Stability and robustness are critical for deploying Transformers in safety-sensitive settings. A principled way to enforce such behavior is to constrain the model's Lipschitz constant. However, approximation-theoretic guarantees for…
The Transformer architecture has revolutionized the field of sequence modeling and underpins the recent breakthroughs in large language models (LLMs). However, a comprehensive mathematical theory that explains its structure and operations…
We rigorously justify the bilayer shallow-water system as an approximation to the hydrostatic Euler equations in situations where the flow is density-stratified with close-to-piecewise constant density profiles, and close-to-columnar…
Transformers have achieved remarkable success across natural language processing (NLP) and computer vision (CV). However, deep transformer models often suffer from an over-smoothing issue, in which token representations converge to similar…
Despite the Transformer's dominance across machine learning, its architecture remains largely heuristic and lacks a unified theoretical foundation. We introduce Score-based Variational Flow (SVFlow), a continuous-time dynamical system for…
Many simulated complex systems that support persistent self-organizing patterns, i.e. gliders, have a 'state-plus-update' paradigm. This approach can be found in computational models of physics, continuous and neural cellular automata,…
Transformer-based models have recently become wildly successful across a diverse set of domains. At the same time, recent work has shown empirically and theoretically that Transformers are inherently limited. Specifically, they argue that…
In this article we consider the multi-layer shallow water system for the propagation of gravity waves in density-stratified flows, with additional terms introduced by the oceanographers Gent and McWilliams in order to take into account…
This paper investigates approximation-theoretic aspects of the in-context learning capability of the transformers in representing a family of noisy linear dynamical systems. Our first theoretical result establishes an upper bound on the…
We present theoretical convergence guarantees for ODE-based generative models, specifically flow matching. We use a pre-trained autoencoder network to map high-dimensional original inputs to a low-dimensional latent space, where a…
We consider compressible fluid flow on an evolving surface with a piecewise Lipschitz-continuous boundary from an energetic point of view. We employ both an energetic variational approach and the first law of thermodynamics to make a…
An implicit Euler finite-volume scheme for general cross-diffusion systems with volume-filling constraints is proposed and analyzed. The diffusion matrix may be nonsymmetric and not positive semidefinite, but the diffusion system is assumed…
We present a hybrid transformer architecture that replaces discrete middle layers with a continuous-depth Neural Ordinary Differential Equation (ODE) block, enabling inference-time control over generation attributes via a learned steering…
Directed percolation (DP), a universality class of continuous phase transitions, has recently been established as a possible route to turbulence in subcritical wall-bounded flows. In canonical straight pipe or planar flows, the transition…
Transformers have achieved state-of-the-art performance in numerous tasks. In this paper, we propose a continuous-time formulation of transformers. Specifically, we consider a dynamical system whose governing equation is parametrized by…
Probabilistic integration of a continuous dynamical system is a way of systematically introducing model error, at scales no larger than errors introduced by standard numerical discretisation, in order to enable thorough exploration of…
We model the evolution of tokens within a deep stack of Transformer layers as a continuous-time flow on the unit sphere, governed by a mean-field interacting particle system, building on the framework introduced in (Geshkovski et al.,…
We investigate the fundamental limits of transformer-based foundation models, extending our analysis to include Visual Autoregressive (VAR) transformers. VAR represents a big step toward generating images using a novel, scalable,…
In spite of their huge success, transformer models remain difficult to scale in depth. In this work, we develop a unified signal propagation theory and provide formulae that govern the moments of the forward and backward signal through the…
The deformation of a viscous liquid droplet suspended in another liquid and subject to an applied electric field is a classic multiphase flow problem best described by the Melcher-Taylor leaky dielectric model. The main assumption of the…