Turbulence-like 5/3 spectral scaling in contextual representations of language as a complex system
Abstract
Natural language is a complex system that exhibits robust statistical regularities. Here, we represent text as a trajectory in a high-dimensional embedding space generated by transformer-based language models, and quantify scale-dependent fluctuations along the token sequence using an embedding-step signal. Across multiple languages and corpora, the resulting power spectrum exhibits a robust power law with an exponent close to over an extended frequency range. This scaling is observed consistently in contextual embeddings from both human-written and AI-generated text, but is absent in static word embeddings and is disrupted by randomization of token order. These results show that the observed scaling reflects multiscale, context-dependent organization rather than lexical statistics alone. By analogy with the Kolmogorov spectrum in turbulence, our findings suggest that semantic information is integrated in a scale-free, self-similar manner across linguistic scales, and provide a quantitative, model-agnostic benchmark for studying complex structure in language representations.
Cite
@article{arxiv.2604.05536,
title = {Turbulence-like 5/3 spectral scaling in contextual representations of language as a complex system},
author = {Zhongxin Yang and Chun Bao and Yuanwei Bin and Xiang I. A. Yang and Shiyi Chen},
journal= {arXiv preprint arXiv:2604.05536},
year = {2026}
}