Transformers learn through gradual rank increase

Enric Boix-Adsera; Etai Littwin; Emmanuel Abbe; Samy Bengio; Joshua Susskind

Transformers learn through gradual rank increase

Machine Learning 2023-12-12 v2

Authors: Enric Boix-Adsera , Etai Littwin , Emmanuel Abbe , Samy Bengio , Joshua Susskind

Abstract

We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs under the simplifying assumptions of diagonal weight matrices and small initialization. Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.

Keywords

transformer machine learning theory

Cite

@article{arxiv.2306.07042,
  title  = {Transformers learn through gradual rank increase},
  author = {Enric Boix-Adsera and Etai Littwin and Emmanuel Abbe and Samy Bengio and Joshua Susskind},
  journal= {arXiv preprint arXiv:2306.07042},
  year   = {2023}
}

Comments

39 pages, to appear in NeurIPS 2023

Related papers

View all related →