English

Two-Scale Latent Dynamics for Recurrent-Depth Transformers

Machine Learning 2025-11-14 v2

Abstract

Recurrent-depth transformers scale test-time compute by iterating latent computations before emitting tokens. We study the geometry of these iterates and argue for a simple, two-scale operational picture: (i) within a looped block, updates act as small-scale refinements; (ii) across consecutive blocks, states undergo a larger-scale drift. Across training, our measurements show that loop steps become smaller and increasingly orthogonal to one another, indicating better local modeling of fine structure rather than merely pushing in a single direction. These dynamics motivate an early-exit mechanism based on the model's second-order difference in step-size, which we show is superior in terms of performance, stability and time-efficiency, when compared to the KL-divergence exit strategy of Geiping et al. and its naive first-order counterpart.

Keywords

Cite

@article{arxiv.2509.23314,
  title  = {Two-Scale Latent Dynamics for Recurrent-Depth Transformers},
  author = {Francesco Pappone and Donato Crisostomi and Emanuele Rodolà},
  journal= {arXiv preprint arXiv:2509.23314},
  year   = {2025}
}
R2 v1 2026-07-01T06:00:52.901Z