Related papers: Uniform Scaling Limits in AdamW-Trained Transforme…
Time series alignment methods call for highly expressive, differentiable and invertible warping functions which preserve temporal topology, i.e diffeomorphisms. Diffeomorphic warping functions can be generated from the integration of…
In this paper, we develop a novel argument, the non-autonomous approximation method, to seek the asymptotic limits of the fully coupled multi-scale McKean-Vlasov stochastic systems with irregular coefficients, which, as summarized in…
In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and…
In a recent paper by Guglielmi and Hairer (SIADS 2015), an analysis in the $\varepsilon\to 0$ limit was proposed of regularized discontinuous ODEs in codimension-2 switching domains; this was obtained by studying a certain 2-dimensional…
In recent years, increasingly large models have achieved outstanding performance across CV tasks. However, these models demand substantial computational resources and storage, and their growing complexity limits our understanding of how…
Stochastic differential equations (SDE) often exhibit large random transitions. This property, which we denote as pathwise stiffness, causes transient bursts of stiffness which limit the allowed step size for common fixed time step explicit…
We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein…
Compressed Deep Learning (DL) models are essential for deployment in resource-constrained environments. But their performance often lags behind their large-scale counterparts. To bridge this gap, we propose Alignment Adapter (AlAd): a…
The multi-head attention layer is one of the key components of the transformer architecture that sets it apart from traditional feed-forward models. Given a sequence length $k$, attention matrices…
Amodal depth estimation aims to predict the depth of occluded (invisible) parts of objects in a scene. This task addresses the question of whether models can effectively perceive the geometry of occluded regions based on visible cues. Prior…
We investigate two one-dimensional tight-binding models with disorder that have extended states at zero energy. We use exact and partial diagonalisation of the Hamiltonian to obtain the eigenmodes and the associated participation ratios,…
Neural operators offer a powerful data-driven framework for learning mappings between function spaces, in which the transformer-based neural operator architecture faces a fundamental scalability-accuracy trade-off: softmax attention…
Augmenting mechanistic ordinary differential equation (ODE) models with machine-learnable structures is an novel approach to create highly accurate, low-dimensional models of engineering systems incorporating both expert knowledge and…
Despite the promise of scientific machine learning (SciML) in combining data-driven techniques with mechanistic modeling, existing approaches for incorporating hard constraints in neural differential equations (NDEs) face significant…
Many important problems in science and engineering require solving the so-called parametric partial differential equations (PDEs), i.e., PDEs with different physical parameters, boundary conditions, shapes of computational domains, etc.…
Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping…
As emerging quantum architectures evolve into heterogeneous networks combining different physical substrates, such as qubits for logic and higher-dimensional qudits for robust communication, the traditional scalar metrics of quantum error…
The canonical $O(N^2)$ Transformer remains the empirical performance frontier in sequence modeling, and its training can be further optimized by addressing geometric inefficiency. We propose an optimization framework that leverages an…
At low temperature T, a significant difference between the behavior of crystals on the one hand and disordered solids on the other is seen: sufficiently strong disorder can give rise to a transition of the transport properties from…
Modern large language models (LLMs) excel at tasks that require storing and retrieving knowledge, such as factual recall and question answering. Transformers are central to this capability because they can encode information during training…