Related papers: Localmax dynamics for attention in transformers an…

This paper studies non-trivial consensus--a relatively novel and unexplored convergence behavior--on directed signed matrix-weighted networks subject to both additive and multiplicative measurement noises under time-varying topologies.…

This paper introduces a novel approach for modeling the dynamics of soft robots, utilizing a differentiable filter architecture. The proposed approach enables end-to-end training to learn system dynamics, noise characteristics, and temporal…

Large Language Models (LLMs) have become the central paradigm in artificial intelligence, yet the core computational primitive of attention has remained structurally unchanged. Local Linear Attention (LLA) is an attention mechanism derived…

Since the introduction of the Transformer architecture for large language models, the softmax-based attention layer has faced increasing scrutinity due to its quadratic-time computational complexity. Attempts have been made to replace it…

We propose a novel flexible-step model predictive control algorithm for unknown linear time-invariant discrete-time systems. The goal is to asymptotically stabilize the system without relying on a pre-collected dataset that describes its…

We study the convergence behavior of the stochastic heavy-ball method with a small stepsize. Under a change of time scale, we approximate the discrete method by a stochastic differential equation that models small random perturbations of a…

This paper deals with a new Tikhonov regularized primal-dual dynamical system with variable mass and Hessian-driven damping for solving a convex optimization problem with linear equality constraints. The system features several…

In order to successfully perform tasks specified by natural language instructions, an artificial agent operating in a visual world needs to map words, concepts, and actions from the instruction to visual elements in its environment. This…

We examine a class of stochastic differential inclusions involving multiscale effects designed to solve a class of generalized variational inequalities. This class of problems contains constrained convex non-smooth optimization problems,…

Gating mechanisms have been widely utilized, from early models like LSTMs and Highway Networks to recent state space models, linear attention, and also softmax attention. Yet, existing literature rarely examines the specific effects of…

Control of multistable dynamical system has important applications, from physics to biology. Here, we attack this problem from the perspective of local sensitivity analysis. We develop sensitivity rules to control properties of…

We study a stochastic optimization problem in which the sampling distribution depends on the decision variable, and the available samples are generated through an iterate-dependent Markov chain. Such settings arise naturally in problems…

The majority of machine learning methods can be regarded as the minimization of an unavailable risk function. To optimize the latter, given samples provided in a streaming fashion, we define a general stochastic Newton algorithm and its…

The attention mechanism is a pivotal element within the transformer architecture, making a substantial contribution to its exceptional performance. Within this attention mechanism, Softmax is an imperative component that enables the model…

In this paper we study a local and a non-local regularization of the system of nonlinear elastodynamics with a non-convex energy. We show that solutions of the non-local model converge to those of the local model in a certain regime. The…

The quadratic computation complexity of self-attention has been a persistent challenge when applying Transformer models to vision tasks. Linear attention, on the other hand, offers a much more efficient alternative with its linear…

First order optimization algorithms play a major role in large scale machine learning. A new class of methods, called adaptive algorithms, were recently introduced to adjust iteratively the learning rate for each coordinate. Despite great…

We study the asymptotic behavior of the trajectory of a nonautonomous evolution equation governed by a quasi-nonexpansive operator in Hilbert spaces. We prove the weak convergence of the trajectory to a fixed point of the operator by…

In this paper, we investigate the consensus problem of second-order multiagent systems under directed graphs. Simple yet robust consensus algorithms that advance existing achievements in accounting for velocity and input constraints, agent…

This paper introduces a novel continuous-time dynamic average consensus algorithm for networks whose interaction is described by a strongly connected and weight-balanced directed graph. The proposed distributed algorithm allows agents to…