Related papers: Attention's forward pass and Frank-Wolfe

We formalise decompression planning as an optimal control problem with gas feasibility windows (ppO$_2$, END), affine ceilings, and convex penalties in normalised oversaturation. The depth trajectory is constrained to be a monotone ascent,…

To improve the robustness of transformer neural networks used for temporal-dynamics prediction of chaotic systems, we propose a novel attention mechanism called easy attention which we demonstrate in time-series reconstruction and…

In the theory of Partially Observed Markov Decision Processes (POMDPs), existence of optimal policies have in general been established via converting the original partially observed stochastic control problem to a fully observed one on the…

Transformers have achieved state-of-the-art performance across various tasks, but suffer from a notable quadratic complexity in sequence length due to the attention mechanism. In this work, we propose MonarchAttention -- a novel approach to…

We propose and analyze a new dynamical system with a closed-loop control law in a Hilbert space $\mathcal{H}$, aiming to shed light on the acceleration phenomenon for \textit{monotone inclusion} problems, which unifies a broad class of…

In this work we study thermodynamic manifestations of the quantum criticality in multiband unconventional superconductors. As a guiding example we consider the scenario of magnetic quantum critical point in the model that captures…

We study a quantum dot coupled to two semiconducting reservoirs, when the dot level and the electrochemical potential are both close to a band edge in the reservoirs. This is modelled with an exactly solvable Hamiltonian without…

Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as…

We propose in this paper New Q-Newton's method. The update rule is very simple conceptually, for example $x_{n+1}=x_n-w_n$ where $w_n=pr_{A_n,+}(v_n)-pr_{A_n,-}(v_n)$, with $A_n=\nabla ^2f(x_n)+\delta _n||\nabla f(x_n)||^2.Id$ and…

We characterize the pre-softmax attention matrix $\mathbf{QK^\top}$ in transformers as an associative memory matrix encoding pairwise associations between input features. By decomposing this matrix into its symmetric and skew-symmetric…

Markov Decision Processes (MDPs) are a formal framework for modeling and solving sequential decision-making problems. In finite-time horizons such problems are relevant for instance for optimal stopping or specific supply chain problems,…

Dot-product attention mechanism plays a crucial role in modern deep architectures (e.g., Transformer) for sequence modeling, however, na\"ive exact computation of this model incurs quadratic time and memory complexities in sequence length,…

We consider an optimization problem over measures for emergency response to out-of-hospital cardiac arrest (OHCA), where the goal is to allocate volunteer resources across a spatial region to minimize the probability of death. The problem…

The Transformer model architecture has become one of the most widely used in deep learning and the attention mechanism is at its core. The standard attention formulation uses a softmax operation applied to a scaled dot product between query…

In the present paper, we investigate the dynamics of a Bose-Einstein condensates (BEC) loaded into an deep optical lattice of 1D, 2D and 3D, both analytically and numerically. We focus on the self-trapping state and the effect of the system…

Probing quantum dynamics in the strong-field regime is critical for advancing our understanding of controlled quantum systems and developing robust quantum technologies. In this work, we experimentally investigate the dynamics of a trapped…

We study the finite-frequency inter-band transition peak in the optical conductivity of a heavy fermion system close to a Kondo breakdown quantum critical point, where the lattice Kondo temperature vanishes. As the system approaches the…

In this paper, we study the episodic reinforcement learning (RL) problem modeled by finite-horizon Markov Decision Processes (MDPs) with constraint on the number of batches. The multi-batch reinforcement learning framework, where the agent…

We study two strange phenomena in auto-regressive Transformers: (1) the dominance of the first token in attention heads; (2) the occurrence of large outlier activations in the hidden states. We find that popular large language models, such…

Transformer architecture has shown impressive performance in multiple research domains and has become the backbone of many neural network models. However, there is limited understanding on how it works. In particular, with a simple…