On the Emergence and Test-Time Use of Structural Information in Large Language Models
Computation and Language
2026-01-27 v1 Machine Learning
Abstract
Learning structural information from observational data is central to producing new knowledge outside the training corpus. This holds for mechanistic understanding in scientific discovery as well as flexible test-time compositional generation. We thus study how language models learn abstract structures and utilize the learnt structural information at test-time. To ensure a controlled setup, we design a natural language dataset based on linguistic structural transformations. We empirically show that the emergence of learning structural information correlates with complex reasoning tasks, and that the ability to perform test-time compositional generation remains limited.
Keywords
Cite
@article{arxiv.2601.17869,
title = {On the Emergence and Test-Time Use of Structural Information in Large Language Models},
author = {Michelle Chao Chen and Moritz Miller and Bernhard Schölkopf and Siyuan Guo},
journal= {arXiv preprint arXiv:2601.17869},
year = {2026}
}