High-quality long-context data is essential for training large language models (LLMs) capable of processing extensive documents, yet existing synthesis approaches using relevance-based aggregation face challenges of computational efficiency. We present LiteLong, a resource-efficient method for synthesizing long-context data through structured topic organization and multi-agent debate. Our approach leverages the BISAC book classification system to provide a comprehensive hierarchical topic organization, and then employs a debate mechanism with multiple LLMs to generate diverse, high-quality topics within this structure. For each topic, we use lightweight BM25 retrieval to obtain relevant documents and concatenate them into 128K-token training samples. Experiments on HELMET and Ruler benchmarks demonstrate that LiteLong achieves competitive long-context performance and can seamlessly integrate with other long-dependency enhancement methods. LiteLong makes high-quality long-context data synthesis more accessible by reducing both computational and data engineering costs, facilitating further research in long-context language training.
@article{arxiv.2509.15568,
title = {LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs},
author = {Junlong Jia and Xing Wu and Chaochen Gao and Ziyang Chen and Zijia Lin and Zhongzhi Li and Weinong Wang and Haotian Xu and Donghui Jin and Debing Zhang and Binghui Guo},
journal= {arXiv preprint arXiv:2509.15568},
year = {2025}
}