Long-context modeling has drawn more and more attention in the area of Large Language Models (LLMs). Continual training with long-context data becomes the de-facto method to equip LLMs with the ability to process long inputs. However, it still remains an open challenge to measure the quality of long-context training data. To address this issue, we propose a Long-context data selection framework with Attention-based Dependency Measurement (LADM), which can efficiently identify high-quality long-context data from a large-scale, multi-domain pre-training corpus. LADM leverages the retrieval capabilities of the attention mechanism to capture contextual dependencies, ensuring a comprehensive quality measurement of long-context data. Experimental results show that our LADM framework significantly boosts the performance of LLMs on multiple long-context tasks with only 1B tokens for continual training.
@article{arxiv.2503.02502,
title = {LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs},
author = {Jianghao Chen and Junhong Wu and Yangyifan Xu and Jiajun Zhang},
journal= {arXiv preprint arXiv:2503.02502},
year = {2025}
}
Comments
ACL 2025, our code is available at https://github.com/ZNLP/LADM