English

Latent Trajectory Dynamics in Large Language Models: A Manifold Evolution Framework with Empirical Validation

Computation and Language 2026-05-05 v3 Artificial Intelligence

Abstract

Understanding how latent representations evolve during generation is a central open problem in large language model interpretability. We introduce \textbf{Dynamical Manifold Evolution Theory} (DMET), a phenomenological framework that models LLM generation as a controlled dynamical system evolving along a trajectory on a low-dimensional semantic manifold. DMET formalizes the structural correspondence between Transformer components and a first-order ODE governed by a semantic potential VV, and characterizes trajectory geometry through three falsifiable proxy metrics: state continuity CC, attractor clustering quality QQ, and topological persistence PP, targeting local smoothness, meso-scale basin structure, and global topological organization, respectively. Across six model architectures, four task types, and 1,080 experimental runs, all three metrics consistently predict text quality outcomes -- log-perplexity, grammaticality, and cross-sentence coherence -- after controlling for decoding parameters, with associations surviving Benjamini--Hochberg correction. Ablation and sanity-check experiments confirm that the effects arise from genuine trajectory structure rather than static distributional artefacts. Furthermore, online monitoring of CC drives an adaptive decoding controller that reduces perplexity from 48.5 to 14.6 relative to a fixed-parameter baseline, demonstrating that latent dynamics characterization translates directly into actionable generation control.

Keywords

Cite

@article{arxiv.2505.20340,
  title  = {Latent Trajectory Dynamics in Large Language Models: A Manifold Evolution Framework with Empirical Validation},
  author = {Yukun Zhang and Qi Dong and Mengkang Li},
  journal= {arXiv preprint arXiv:2505.20340},
  year   = {2026}
}
R2 v1 2026-07-01T02:40:44.138Z