Related papers: Attention in Krylov Space
Krylov complexity is considered to provide a measure of the growth of operators evolving under Hamiltonian dynamics. The main strategy is the analysis of the structure of Krylov subspace $\mathcal{K}_M(\mathcal{H},\eta)$ spanned by the…
In many real-world scenarios, data to train machine learning models becomes available over time. Unfortunately, these models struggle to continually learn new concepts without forgetting what has been learnt in the past. This phenomenon is…
Large-scale foundation models for scientific machine learning adapt to physical settings unseen during training, such as zero-shot transfer between turbulent scales. This phenomenon, in-context learning, challenges conventional…
Operator learning based on neural operators has emerged as a promising paradigm for the data-driven approximation of operators, mapping between infinite-dimensional Banach spaces. Despite significant empirical progress, our theoretical…
Adaptive control of Euler-Lagrange systems is challenging when friction is governed by a finite-horizon internal state that is not directly observable from joint measurements. In this setting, the measured closed-loop state is no longer…
Data-driven learning of partial differential equations' solution operators has recently emerged as a promising paradigm for approximating the underlying solutions. The solution operators are usually parameterized by deep learning models…
Approximating the action of a matrix function $f(\mathbf{A})$ on a vector $\mathbf{b}$ is an increasingly important primitive in machine learning, data science, and statistics, with applications such as sampling high dimensional Gaussians,…
We increase the efficiency of a recently proposed time integration scheme for time dependent quantum transport by using the Lanczos method for time evolution. We illustrate our modified scheme in terms of a simple one dimensional model. Our…
We study Krylov complexity in Lifshitz-type Dirac field theories with a generic dynamical critical exponent $z$. By computing the Lanczos coefficients for massless and massive cases, we analyze the growth and saturation behavior of Krylov…
Convolutional analysis operator learning (CAOL) enables the unsupervised training of (hierarchical) convolutional sparsifying operators or autoencoders from large datasets. One can use many training images for CAOL, but a precise…
Transformer is a ubiquitous model for natural language processing and has attracted wide attentions in computer vision. The attention maps are indispensable for a transformer model to encode the dependencies among input tokens. However,…
It was proposed recently that the out-of-time-ordered four-point correlator (OTOC) may serve as a useful characteristic of quantum-chaotic behavior, because in the semi-classical limit, $\hbar \to 0$, its rate of exponential growth…
In an isolated system, the time evolution of a given observable in the Heisenberg picture can be efficiently represented in Krylov space. In this representation, an initial operator becomes increasingly complex as time goes by, a feature…
The self-attention mechanism is the key to the success of transformers in recent Large Language Models (LLMs). However, the quadratic computational cost $O(n^2)$ in the input sequence length $n$ is a notorious obstacle for further…
Attention mechanisms have significantly advanced visual models by capturing global context effectively. However, their reliance on large-scale datasets and substantial computational resources poses challenges in data-scarce and…
Krylov complexity measures operator growth with respect to a basis, which is adapted to the Heisenberg time evolution. The construction of that basis relies on the Lanczos algorithm, also known as the recursion method. The mathematics of…
We investigate various aspects of the Lanczos coefficients in a family of free Lifshitz scalar theories, characterized by their integer dynamical exponent, at finite temperature. In this non-relativistic setup, we examine the effects of…
We study the statistical properties of Lanczos coefficients over an ensemble of random initial operators generating the Krylov space. We propose two statistical quantities that are important in characterizing the complexity: the average…
Transformers have achieved state-of-the-art results across a range of domains, but their quadratic attention mechanism poses significant challenges for long-sequence modelling. Recent efforts to design linear-time attention mechanisms have…
Although Transformers with fully connected self-attentions are powerful to model long-term dependencies, they are struggling to scale to long texts with thousands of words in language modeling. One of the solutions is to equip the model…