Related papers: Localmax dynamics for attention in transformers an…

Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight…

The original softmax-based attention mechanism (regular attention) in the extremely successful Transformer architecture computes attention between $N$ tokens, each embedded in a $D$-dimensional head, with a time complexity of $O(N^2D)$.…

Transformer self-attention can be interpreted as a gradient flow on the unit sphere, in which tokens evolve under softmax interaction potentials and tend to form clusters. While prior work has established clustering behavior for single-head…

We study the problem of solving fixed-point equations for seminorm-contractive operators and establish foundational results on the non-asymptotic behavior of iterative algorithms in both deterministic and stochastic settings. Specifically,…

This paper develops new techniques for studying smooth dynamical systems in the presence of a \CC metric. Principally, we employ the theory of Margulis-Mostow, M\'etivier, Mitchell and Pansu on tangent cones to establish resonances between…

This paper provides a systematic exposition of Lyapunov stability for compact sets in locally compact metric spaces. We explore foundational concepts, including neighborhoods of compact sets, invariant sets, and the properties of dynamical…

This paper provides a new unified framework for second-moment stability of discrete-time linear systems with stochastic dynamics. Relations of notions of second-moment stability are studied for the systems with general stochastic dynamics,…

This work investigates a dynamical system functioning as a nonsmooth adaptation of the continuous Newton method, aimed at minimizing the sum of a primal lower-regular and a locally Lipschitz function, both potentially nonsmooth. The…

In this paper, we present new results on finite- and fixed-time convergence for dynamical systems using LaSalle-like invariance principles. In particular, we provide first and second-order non-smooth Lyapunov-like results for finite- and…

It has been challenging to model the complex temporal-spatial dependencies between agents for trajectory prediction. As each state of an agent is closely related to the states of adjacent time steps, capturing the local temporal dependency…

We study asymptotic stability of continuous-time systems with mode-dependent guaranteed dwell time. These systems are reformulated as special cases of a general class of mixed (discrete-continuous) linear switching systems on graphs, in…

While linear attention reduces the quadratic complexity of standard Transformers to linear time, it often lags behind in expressivity due to the removal of softmax normalization. This omission eliminates \emph{global competition}, a…

We study fundamental performance limitations of distributed feedback control in large-scale networked dynamical systems. Specifically, we address the question of whether dynamic feedback controllers perform better than static (memoryless)…

The attention mechanism is the computational core of modern Transformer architectures, but its quadratic complexity in the input sequence length is the bottleneck for large-scale inference. This has motivated a rapidly growing body of work…

Transformers are mostly relying on softmax attention, which introduces quadratic complexity with respect to sequence length and remains a major bottleneck for efficient inference. Prior work on linear or hybrid attention typically replaces…

The adversarial robustness of attributions is a fundamental requirement for reliable explainability in deep learning, yet existing approaches typically rely on computationally expensive explicit regularization. In this work, we show that…

In this paper, we propose a second-order continuous primal-dual dynamical system with time-dependent positive damping terms for a separable convex optimization problem with linear equality constraints. By the Lyapunov function approach, we…

We study the behavior of classical dimer coverings of the square lattice - a paradigmatic model for systems subject to constraints - evolving under local stochastic dynamics, by means of Monte Carlo simulations and theoretical arguments. We…

The quadratic complexity of the attention mechanism represents one of the biggest hurdles for processing long sequences using Transformers. Current methods, relying on sparse representations or stateful recurrence, sacrifice token-to-token…

This paper is concerned with relationships of Lyapunov exponents with sensitivity and stability for non-autonomous discrete systems. Some new concepts are introduced for non-autonomous discrete systems, including Lyapunov exponents, strong…