Dynamical Behaviors of the Gradient Flows for In-Context Learning
Dynamical Systems
2024-12-24 v1
Abstract
We derive the system of differential equations for the gradient flow characterizing the training process of linear in-context learning in full generality. Next, we explore the geometric structure of the gradient flows in two instances, including identifying its invariants, optimum, and saddle points. This understanding allows us to quantify the behavior of the two gradient flows under the full generality of parameters and data.
Cite
@article{arxiv.2412.16683,
title = {Dynamical Behaviors of the Gradient Flows for In-Context Learning},
author = {Songtao Lu and Yingdong Lu and Tomasz Nowicki},
journal= {arXiv preprint arXiv:2412.16683},
year = {2024}
}