Related papers: Attention's forward pass and Frank-Wolfe
In this paper, we study active set identification results for the away-step Frank-Wolfe algorithm in different settings. We first prove a local identification property that we apply, in combination with a convergence hypothesis, to get an…
By using a different quantum-to-classical mapping from the Trotter-Suzuki decomposition, we identify the entanglement structure of the maximal eigenvectors for the associated quantum transfer matrix. This observation provides a deeper…
Transformer-based models have demonstrated exceptional performance across diverse domains, becoming the state-of-the-art solution for addressing sequential machine learning problems. Even though we have a general understanding of the…
The most widely used artificial intelligence (AI) models today are Transformers employing self-attention. In its standard form, self-attention incurs costs that increase with context length, driving demand for storage, compute, and energy…
We present a novel local Lipschitz bound for self-attention blocks of transformers. This bound is based on a refined closed-form expression for the spectral norm of the softmax function. The resulting bound is not only more accurate than in…
Recent work has shown that in certain settings, the Frank-Wolfe algorithm (FW) with open-loop step-sizes $\eta_t = \frac{\ell}{t+\ell}$ for a fixed parameter $\ell \in \mathbb{N},\, \ell \geq 2$, attains a convergence rate faster than the…
Motivated by applications in Game Theory, Optimization, and Generative Adversarial Networks, recent work of Daskalakis et al \cite{DISZ17} and follow-up work of Liang and Stokes \cite{LiangS18} have established that a variant of the widely…
In this work, we study the asymptotic randomness of an algorithmic estimator of the saddle point of a globally convex-concave and locally strongly-convex strongly-concave objective. Specifically, we show that the averaged iterates of a…
We explore the dynamics of active elements performing persistent random motion with fluctuating active speed and in the presence of translational noise in a $d$-dimensional harmonic trap, modeling active speed generation through an…
In machine learning, a self-attention dynamics is a continuous-time multiagent-like model of the attention mechanisms of transformers. In this paper we show that such dynamics is related to a multiagent version of the Oja flow, a dynamical…
The isotropic-to-nematic transition in a two-dimensional fluid of hard needles is studied using grand canonical Monte Carlo simulations, multiple histogram reweighting, and finite size scaling. The transition is shown to be of the…
It is known that the Frank-Wolfe (FW) algorithm, which is affine-covariant, enjoys accelerated convergence rates when the constraint set is strongly convex. However, these results rely on norm-dependent assumptions, usually incurring…
We present closed-form analytic solutions to non-secular Bloch-Redfield master equations for quantum dynamics of a V-type system driven by weak coupling to a thermal bath. We focus on noise-induced Fano coherences among the excited states…
We perform large-scale Quantum Monte Carlo (QMC) simulations for strongly interacting bosons in a 2D optical lattice trap, and confirm an excellent agreement with the benchmarking in-situ density measurements by the Chicago group [1]. We…
We study the current, the curvature of levels, and the finite temperature charge stiffness, D(T,L), in the strongly correlated limit, U>>t, for Hubbard rings of L sites, with U the on-site Coulomb repulsion and t the hopping integral. Our…
We consider the exclusion process in the one-dimensional discrete torus with $N$ points, where all the bonds have conductance one, except a finite number of slow bonds, with conductance $N^{-\beta}$, with $\beta\in[0,\infty)$. We prove that…
A fundamental result in modern quantum chaos theory is the Maldacena-Shenker-Stanford upper bound on the growth of out-of-time-order correlators, whose infinite-temperature limit is related to the operator-space entanglement entropy of the…
The Frank-Wolfe algorithm is a popular method in structurally constrained machine learning applications, due to its fast per-iteration complexity. However, one major limitation of the method is a slow rate of convergence that is difficult…
The quadratic complexity of the attention mechanism represents one of the biggest hurdles for processing long sequences using Transformers. Current methods, relying on sparse representations or stateful recurrence, sacrifice token-to-token…
We investigate the nonlinear Bloch dynamics and Landau-Zener tunneling of quantum droplets in optical lattices, where the interplay between mean-field repulsion and beyond-mean-field attraction from Lee-Huang-Yang corrections introduces a…