中文

LayerNorm as Implicit Gain Control in Looped Transformers

机器学习 2026-07-12 v1 神经与进化计算

摘要

In pre-LayerNorm looped transformers, LayerNorm inside the recurrent block acts as an implicit gain controller: by coupling the block's local Lipschitz constant inversely to the activation scale, it renders the recurrence Jacobian non-normal -- asymptotically contractive at every verified fixed point even where its operator norm exceeds 1 -- so the true stability budget is the spectral margin, not an operator-norm bound. That margin depletes as the carry ρ1\rho \to 1, and a minority of initializations never converge to a fixed point at all, so the diagonal carry constraint ρ(Aˉ)<1\rho(\bar{A}) < 1 is necessary but not sufficient for convergence of the full recurrence. Training experiments across six tasks, including a controlled ablation, reveal that the linear carry is not the depth-memory mechanism: gradient descent routes memory through the block's more expressive nonlinear recurrence and leaves the stability-constrained carry at rest -- the carry's role is stabilization, not memory. We characterize the boundary of this claim: on tasks with axis-aligned per-channel structure, gradient descent does recruit the carry. All results are derived analytically and verified in a from-scratch, CPU-scale implementation; verification at larger scale is needed.

引用

@article{arxiv.2607.10681,
  title  = {LayerNorm as Implicit Gain Control in Looped Transformers},
  author = {Matthias M. M. Buehlmaier},
  journal= {arXiv preprint arXiv:2607.10681},
  year   = {2026}
}

备注

23 pages, 6 figures, 13 tables. Code available at https://github.com/clojure-finance/mythjure/tree/paper-v1