In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Tasks
Machine Learning
2026-02-20 v1 Artificial Intelligence
Abstract
Recent work has demonstrated that transformers and linear attention models can perform in-context learning (ICL) on simple function classes, such as linear regression. In this paper, we empirically study how these two attention mechanisms differ in their ICL behavior on the canonical linear-regression task of Garg et al. We evaluate learning quality (MSE), convergence, and generalization behavior of each architecture. We also analyze how increasing model depth affects ICL performance. Our results illustrate both the similarities and limitations of linear attention relative to quadratic attention in this setting.
Cite
@article{arxiv.2602.17171,
title = {In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Tasks},
author = {Ayush Goel and Arjun Kohli and Sarvagya Somvanshi},
journal= {arXiv preprint arXiv:2602.17171},
year = {2026}
}