Related papers: Dynamic metastability in the self-attention model

We introduce a novel model for active particles with short-range aligning interactions and study their behaviour in crowded environments using numerical simulations. When only active particles are present, we observe a transition from a…

We use coarse-grained molecular dynamics simulations to study the motility of a 2D vesicle containing self-propelled rods, as a function of the vesicle bending rigidity and the number density, length, and activity of the enclosed rods.…

We investigate the dynamics of a deterministic self-propelled particle endowed with coherent memory. We evidence experimentally and numerically that it exhibits several stable free states. The system is composed of a self-propelled drop…

We discuss a simple model of particles hopping in one dimension with attractive interactions. Taking a hydrodynamic limit in which the interaction strength increases with the system size, we observe the formation of multiple clusters of…

We study the connection between gradient-based meta-learning and convex op-timisation. We observe that gradient descent with momentum is a special case of meta-gradients, and building on recent results in optimisation, we prove convergence…

In a Newtonian system with localized interactions the whole set of particles is naturally decomposed into dynamical clusters, defined as finite groups of particles having an influence on each other's trajectory during a given interval of…

We explore the dynamics of active elements performing persistent random motion with fluctuating active speed and in the presence of translational noise in a $d$-dimensional harmonic trap, modeling active speed generation through an…

Self-attention has emerged as a core component of modern neural architectures, yet its theoretical underpinnings remain elusive. In this paper, we study self-attention through the lens of interacting entities, ranging from agents in…

In spite of their huge success, transformer models remain difficult to scale in depth. In this work, we develop a unified signal propagation theory and provide formulae that govern the moments of the forward and backward signal through the…

Transformer-based language models are trained on large datasets to predict the next token given an input sequence. Despite this simple training objective, they have led to revolutionary advances in natural language processing. Underlying…

We analyze the dynamics of concentrated polymer solutions modeled by a 2D Smoluchowski equation. We describe the long time behavior of the polymer suspensions in a fluid. When the flow influence is neglected the equation has a gradient…

Transformers have emerged as a powerful tool for a broad range of natural language processing tasks. A key component that drives the impressive performance of Transformers is the self-attention mechanism that encodes the influence or…

As artificial intelligence models have exploded in scale and capability, understanding of their internal mechanisms remains a critical challenge. Inspired by the success of dynamical systems approaches in neuroscience, here we propose a…

In this study, we present results from experiments on the retention of single oil droplets rising through a two-layer density stratification. These experiments confirm the significant slowdown observed in past literature of settling and…

The dynamic of complex ordering systems with active rotational degrees of freedom exemplified by protein self-assembly is explored using a machine learning workflow that combines deep learning-based semantic segmentation and rotationally…

The incredible success of transformers on sequence modeling tasks can be largely attributed to the self-attention mechanism, which allows information to be transferred between different parts of a sequence. Self-attention allows…

We study the collective dynamics of repulsive self-propelled particles. The particles are governed by coupled equations of motion that include polar self-propulsion, damping of velocity and of polarity, repulsive particle-particle…

The attention mechanism has been widely used in deep neural networks as a model component. By now, it has become a critical building block in many state-of-the-art natural language models. Despite its great success established empirically,…

Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long…

Most cells possess the capacity to locomote. Alone or collectively, this allows them to adapt, to rearrange, and to explore their surroundings. The biophysical characterization of such motile processes, in health and disease, has so far…