相关论文: Attention's forward pass and Frank-Wolfe
The self-attention mechanism has significantly advanced the field of natural language processing, facilitating the development of advanced language-learning machines. Although its utility is widely acknowledged, the precise mechanisms of…
We study the ferromagnetic phase transition in a randomly layered Heisenberg magnet using large-scale Monte-Carlo simulations. Our results provide numerical evidence for the infinite-randomness scenario recently predicted within a…
This paper considers random access in deadline-constrained broadcasting with frame-synchronized traffic. To enhance the maximum achievable timely delivery ratio (TDR), we define a dynamic control scheme that allows each active node to…
We numerically study the entanglement dynamics of free fermions on a cubic lattice with potential disorder following a quantum quench. We focus, in particular, on the metal-insulator transition at a critical disorder strength and compare…
A new extension of the attractive Hubbard model is constructed to study the critical behavior near a finite temperature superconducting phase transition in two dimensions using the recently developed meron-cluster algorithm. Unlike previous…
Vision Transformers have achieved impressive performance in video classification, while suffering from the quadratic complexity caused by the Softmax attention mechanism. Some studies alleviate the computational costs by reducing the number…
Text-to-image generation models, especially Multimodal Diffusion Transformers (MMDiT), have shown remarkable progress in generating high-quality images. However, these models often face significant computational bottlenecks, particularly in…
We employ the numerical linked-cluster expansion to study finite-temperature properties of the uniform cubic lattice Hubbard model in the thermodynamic limit for a wide range of interaction strengths and densities. We carry out the…
This paper tackles the growing issue of excessive data transmission in networks. With increasing traffic, backhaul links and core networks are under significant traffic, leading to the investigation of caching solutions at edge routers.…
Transformers and deep state space models (SSMs) sit at opposite ends of a basic design choice: attention routes each query through a growing key-value (KV) cache by content-based matching at quadratic cost, while deep SSMs compress context…
We studied the lock-in transition of charge-density waves of one-dimensional conductors. Though this phenomenon has been known for decades, there are still discrepancies between the theories and the experiments. We focused on the pioneering…
In this paper, Thermofield Dynamics (TFD) is applied to map a quantum optics nonlinear master equation into a Schrodinger-like equation for any arbitrary initial condition. This formalism provides a more efficient way for solving open…
Understanding whether attention mechanisms converge to the kernel regime is foundational to the validity of influence functions for transformer accountability. Exact NTK characterization of softmax attention is precluded by its exponential…
Recent papers have shown that the Frank-Wolfe algorithm (FW) with open-loop step-sizes exhibits rates of convergence faster than the iconic $\mathcal{O}(t^{-1})$ rate. In particular, when the minimizer of a strongly convex function over a…
We study theoretically the influence of the temperature and disorder on the spin wave spectrum of the magnonic crystal Fe$_{1-c}$Co$_{c}$. Our formalism is based on the analysis of a Heisenberg Hamiltonian by means of the wave vector and…
Self-attention mechanisms have revolutionised deep learning architectures, yet their core mathematical structures remain incompletely understood. In this work, we develop a category-theoretic framework focusing on the linear components of…
Despite their power, Transformers face challenges with long sequences due to the quadratic complexity of self-attention. To address this limitation, methods like $k$-Nearest-Neighbor ($k$NN) attention have been introduced [Roy, Saffar,…
We explore the structures of light cone and Regge limit singularities of $n$-point Virasoro conformal blocks in $c>1$ two-dimensional conformal field theories with no chiral primaries, using fusion matrix approach. These CFTs include not…
When a learning algorithm reshapes the data distribution it trains on, the long-run behavior depends on the joint evolution of the policy, the value estimate, and the data distribution. We study finite-state actor-critic mean dynamics on…
The vortex dynamics in a low temperature superconductor deposited on top of a rectangular array of micrometer size permalloy triangles is investigated experimentally. The rectangular unit cell is such that neighboring triangles physically…