LayerNorm as Implicit Gain Control in Looped Transformers
Abstract
In pre-LayerNorm looped transformers, LayerNorm inside the recurrent block acts as an implicit gain controller: by coupling the block's local Lipschitz constant inversely to the activation scale, it renders the recurrence Jacobian non-normal -- asymptotically contractive at every verified fixed point even where its operator norm exceeds 1 -- so the true stability budget is the spectral margin, not an operator-norm bound. That margin depletes as the carry , and a minority of initializations never converge to a fixed point at all, so the diagonal carry constraint is necessary but not sufficient for convergence of the full recurrence. Training experiments across six tasks, including a controlled ablation, reveal that the linear carry is not the depth-memory mechanism: gradient descent routes memory through the block's more expressive nonlinear recurrence and leaves the stability-constrained carry at rest -- the carry's role is stabilization, not memory. We characterize the boundary of this claim: on tasks with axis-aligned per-channel structure, gradient descent does recruit the carry. All results are derived analytically and verified in a from-scratch, CPU-scale implementation; verification at larger scale is needed.
Cite
@article{arxiv.2607.10681,
title = {LayerNorm as Implicit Gain Control in Looped Transformers},
author = {Matthias M. M. Buehlmaier},
journal= {arXiv preprint arXiv:2607.10681},
year = {2026}
}
Comments
23 pages, 6 figures, 13 tables. Code available at https://github.com/clojure-finance/mythjure/tree/paper-v1