相关论文: The Diffusion-Attention Connection

Attention mechanisms are a central property of cognitive systems allowing them to selectively deploy cognitive resources in a flexible manner. Attention has been long studied in the neurosciences and there are numerous phenomenological…

Linear attention has emerged as a promising alternative to softmax-based attention, leveraging kernelized feature maps to reduce complexity from quadratic to linear in sequence length. However, the non-negative constraint on feature maps…

The understanding of cross-field transport is crucial in order to optimize the properties of magnetic confinement in modern fusion devices. In this work, a two-dimensional, simplified model is used to study the turbulent dynamics in the…

In this note, we connect two seemingly unrelated objects: On the one hand is a two-dimensional drift-diffusion process $X$ with divergence-free and time-independent drift $b$. The drift is given by a stationary Gaussian ensemble, and we…

Transformer architectures deliver state-of-the-art accuracy via dense full-attention, but their quadratic time and memory complexity with respect to sequence length limits practical deployment. Linear attention mechanisms offer linear or…

Astrophysical fluids are generically turbulent and this must be taken into account for most transport processes. We discuss how the preexisting turbulence modifies magnetic reconnection and how magnetic reconnection affects the MHD…

In this work, surface diffusion is studied with a different perspective by showing how the corresponding open dynamics is transformed when passing, in a continuous and smooth way, from a pure quantum regime to a full classical regime; the…

Diffusion models have demonstrated remarkable capabilities in synthesizing realistic images, spurring interest in using their representations for various downstream tasks. To better understand the robustness of these representations, we…

We present AttentionBender, a tool that manipulates cross-attention in Video Diffusion Transformers to help artists probe the internal mechanics of black-box video generation. While generative outputs are increasingly realistic, prompt-only…

Diffusion maps (DMAP) are often used as a dimensionality-reduction tool, but more precisely they provide a spectral representation of the intrinsic geometry rather than a complete charting method. To illustrate this distinction, we study a…

Machine learning methods are emerging as a universal paradigm for constructing correlative structure-property relationships in materials science based on multimodal characterization. However, this necessitates development of methods for…

Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction. At the core of the Transformer is the…

Propagation of transition fronts in models of coupled oscillators with non-degenerate on-site potential is usually considered in terms of travelling waves. We show that the system dynamics can be reformulated as an implicit map structure,…

The Transformer is a sequence model that forgoes traditional recurrent architectures in favor of a fully attention-based approach. Besides improving performance, an advantage of using attention is that it can also help to interpret a model…

The paper studies the relationship between diffraction and dynamics for uniformly discrete ergodic point processes in real spaces. This relationship takes the form of an isometric embedding of two L^2 spaces. Diffraction (or equivalently…

Driven surface diffusion occurs, for example, in molecular beam epitaxy when particles are deposited under an oblique angle. Elastic phase transitions happen when normal modes in crystals become soft due to the vanishing of certain elastic…

Time series forecasting is widely used in the fields of equipment life cycle forecasting, weather forecasting, traffic flow forecasting, and other fields. Recently, some scholars have tried to apply Transformer to time series forecasting…

Diffusion-driven patterns appear on curved surfaces in many settings, initiated by unstable modes of an underlying Laplacian operator. On a flat surface or perfect sphere, the patterns are degenerate, reflecting translational/rotational…

Adaptive networks are well-suited to perform decentralized information processing and optimization tasks and to model various types of self-organized and complex behavior encountered in nature. Adaptive networks consist of a collection of…

The Transformer architecture has revolutionized the field of sequence modeling and underpins the recent breakthroughs in large language models (LLMs). However, a comprehensive mathematical theory that explains its structure and operations…