English

Gap-K%: Measuring Top-1 Prediction Gap for Detecting Pretraining Data

Machine Learning 2026-01-29 v1 Artificial Intelligence Computation and Language

Abstract

The opacity of massive pretraining corpora in Large Language Models (LLMs) raises significant privacy and copyright concerns, making pretraining data detection a critical challenge. Existing state-of-the-art methods typically rely on token likelihoods, yet they often overlook the divergence from the model's top-1 prediction and local correlation between adjacent tokens. In this work, we propose Gap-K%, a novel pretraining data detection method grounded in the optimization dynamics of LLM pretraining. By analyzing the next-token prediction objective, we observe that discrepancies between the model's top-1 prediction and the target token induce strong gradient signals, which are explicitly penalized during training. Motivated by this, Gap-K% leverages the log probability gap between the top-1 predicted token and the target token, incorporating a sliding window strategy to capture local correlations and mitigate token-level fluctuations. Extensive experiments on the WikiMIA and MIMIR benchmarks demonstrate that Gap-K% achieves state-of-the-art performance, consistently outperforming prior baselines across various model sizes and input lengths.

Keywords

Cite

@article{arxiv.2601.19936,
  title  = {Gap-K%: Measuring Top-1 Prediction Gap for Detecting Pretraining Data},
  author = {Minseo Kwak and Jaehyung Kim},
  journal= {arXiv preprint arXiv:2601.19936},
  year   = {2026}
}

Comments

under review; 13 pages

R2 v1 2026-07-01T09:22:47.122Z