相关论文: The Mean-Field Dynamics of Transformers

A pathwise large deviation principle in the Wasserstein topology and a pathwise central limit theorem are proved for the empirical measure of a mean-field system of interacting diffusions. The coefficients are path-dependent. The framework…

There has been recent progress on the problem of inferring the structure of interactions in complex networks when they are in stationary states satisfying detailed balance, but little has been done for non-equilibrium systems. Here we…

Detailed mean field and Monte Carlo studies of the dynamic magnetization-reversal transition in the Ising model in its ordered phase under a competing external magnetic field of finite duration have been presented here. Approximate…

Transformers and their attention mechanism have been revolutionary in the field of Machine Learning. While originally proposed for the language data, they quickly found their way to the image, video, graph, etc. data modalities with various…

Attention is fundamental to cognition, yet it remains a challenge to understand attention in tasks approaching real-world complexity. Here, we approached this problem by modeling gaze patterns of monkeys playing Pac-Man. We first show a…

The Transformer and its variants have been proven to be efficient sequence learners in many different domains. Despite their staggering success, a critical issue has been the enormous number of parameters that must be trained (ranging from…

Over the past few decades, the hydrology community has witnessed notable advancements in streamflow prediction, particularly with the introduction of cutting-edge machine-learning algorithms. Recurrent neural networks, especially Long…

Entropy modeling is a key component for high-performance image compression algorithms. Recent developments in autoregressive context modeling helped learning-based methods to surpass their classical counterparts. However, the performance of…

Clustering is the propensity of nodes that share a common neighbour to be connected. It is ubiquitous in many networks but poses many modelling challenges. Clustering typically manifests itself by a higher than expected frequency of…

Transformer architectures deliver state-of-the-art accuracy via dense full-attention, but their quadratic time and memory complexity with respect to sequence length limits practical deployment. Linear attention mechanisms offer linear or…

This study proposes a hybrid model based on Transformers, named MSCMHMST, aimed at addressing key challenges in traffic flow prediction. Traditional single-method approaches show limitations in traffic prediction tasks, whereas hybrid…

We propose Joint MLP/Attention (JoMA) dynamics, a novel mathematical framework to understand the training procedure of multilayer Transformer architectures. This is achieved by integrating out the self-attention layer in Transformers,…

A natural way to characterize the cluster structure of a dataset is by finding regions containing a high density of data. This can be done in a nonparametric way with a kernel density estimate, whose modes and hence clusters can be found…

We develop a systematic typical medium dynamical cluster approximation that provides a proper description of the Anderson localization transition in three dimensions (3D). Our method successfully captures the localization phenomenon both in…

Metastable quantum dynamics of an asymmetric triangular cluster that is coupled to a reservoir is investigated. The dynamics is governed by bath-mediated transitions, which in part require a thermal activation process. The decay rate is…

Recent works have shown that transformers can solve contextual reasoning tasks by internally executing computational graphs called circuits. Circuits often use attention to logically match information from subspaces of the representation,…

Surface reconstruction from raw point clouds has been studied for decades in the computer graphics community, which is highly demanded by modeling and rendering applications nowadays. Classic solutions, such as Poisson surface…

We offer a theoretical mathematical background through Lagrangian optimization on the unit hyperspherical manifold and its tangential collection with application to the Transformer and its token space. Our methods are catered to the…

Recent advancements in attention mechanisms have replaced recurrent neural networks and its variants for machine translation tasks. Transformer using attention mechanism solely achieved state-of-the-art results in sequence modeling. Neural…

In this talk I review some recent developments which shed light on the main connections between structural glasses and mean-field spin glass models with a discontinuous transition. I also discuss the role of quantum fluctuations on the…