相关论文: Attention in Krylov Space

We study how multi-head softmax attention models are trained to perform in-context learning on linear data. Through extensive empirical experiments and rigorous theoretical analysis, we demystify the emergence of elegant attention patterns:…

An a posteriori estimate for the error of a standard Krylov approximation to the matrix exponential is derived. The estimate is based on the defect (residual) of the Krylov approximation and is proven to constitute a rigorous upper bound on…

Despite their central role in the success of foundational models and large-scale language modeling, the theoretical foundations governing the operation of Transformers remain only partially understood. Contemporary research has largely…

We study Krylov complexity in BMN Plane Wave Matrix Model at large mass deformation. We consider various consistent reductions of the matrix model that allow us to perform a Hamiltonian analysis which leads to different notions of the…

Partially Observable Markov Decision Processes (POMDPs) remain a core challenge in reinforcement learning due to incomplete state information. We address this by reformulating POMDPs as fully observable processes with fixed-length…

Neural operators provide fast surrogate models for time-dependent partial differential equations, but their standard autoregressive use usually assumes that the instantaneous field $u(t,\cdot)$ is a complete state. This assumption fails for…

The out-of-time-ordered correlator has been proposed as an indicator of chaos in quantum systems due to its simple interpretation in the semiclassical limit. In particular, its rate of possible exponential growth at $\hbar \to 0$ is closely…

Fixed-point equations with Lipschitz operators have been studied for more than a century, and are central to problems in mathematical optimization, game theory, economics, and dynamical systems, among others. When the Lipschitz constant of…

Krylov subspace methods in quantum dynamics identify the minimal subspace in which a process unfolds. To date, their use is restricted to time evolutions governed by time-independent generators. We introduce a generalization valid for…

Neural operators, as an efficient surrogate model for learning the solutions of PDEs, have received extensive attention in the field of scientific machine learning. Among them, attention-based neural operators have become one of the…

Many quantum gravitational frameworks, such as DBI inflation, k-essence, and effective field theories obtained by integrating out heavy modes, can lead to a non-trivial sound speed. Meanwhile, our universe can be described as an open…

Via operator theoretic methods, we formalize the concentration phenomenon for a given observable `$r$' of a discrete time Markov chain with `$\mu_{\pi}$' as invariant ergodic measure, possibly having support on an unbounded state space. The…

Much of model-based reinforcement learning involves learning a model of an agent's world, and training an agent to leverage this model to perform a task more efficiently. While these models are demonstrably useful for agents, every…

Neural operators have emerged as powerful tools for learning solution operators of partial differential equations. However, in time-dependent problems, standard training strategies such as teacher forcing introduce a mismatch between…

Longitudinal analysis is important in many disciplines, such as the study of behavioral transitions in social science. Only very recently, feature selection has drawn adequate attention in the context of longitudinal modeling. Standard…

Several non-linear operators in stochastic analysis, such as solution maps to stochastic differential equations, depend on a temporal structure which is not leveraged by contemporary neural operators designed to approximate general maps…

The mathematical properties and data-driven learning of the Koopman operator, which represents nonlinear dynamics as a linear mapping on a properly defined functional spaces, have become key problems in nonlinear system identification and…

While many problems in machine learning focus on learning mappings between finite-dimensional spaces, scientific applications require approximating mappings between function spaces, i.e., operators. We study the problem of learning…

Operator growth, or operator spreading, describes the process where a "simple" operator acquires increasing complexity under the Heisenberg time evolution of a chaotic dynamics, therefore has been a key concept in the study of quantum chaos…

Nonlinear dynamical systems are ubiquitous in nature and they are hard to forecast. Not only they may be sensitive to small perturbations in their initial conditions, but they are often composed of processes acting at multiple scales.…