English

The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression

Machine Learning 2026-04-10 v1

Abstract

We decompose the spectral edge -- the dominant direction of the Gram matrix of parameter updates -- into its gradient and weight-decay components during grokking in two sequence tasks (Dyck-1 and SCAN). We find a sharp two-phase lifecycle: before grokking the edge is gradient-driven and functionally active; at grokking, gradient and weight decay align, and the edge becomes a compression axis that is perturbation-flat yet ablation-critical (>4000x more impactful than random directions). Three universality classes emerge (functional, mixed, compression), predicted by the gap flow equation. Nonlinear probes show information is re-encoded, not lost (MLP R2=0.99R^2=0.99 where linear R2=0.86R^2=0.86), and removing weight decay post-grok reverses compression while preserving the algorithm.

Keywords

Cite

@article{arxiv.2604.07380,
  title  = {The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression},
  author = {Yongzhong Xu},
  journal= {arXiv preprint arXiv:2604.07380},
  year   = {2026}
}

Comments

15 pages, 12 figures