Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
Abstract
Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writing, with these limitations largely attributed to teacher-forced training. Multi-token prediction (MTP) partially mitigates these issues by predicting several future tokens at once, but it mostly captures short-range dependencies and offers limited improvement. We propose future summary prediction (FSP), which trains an auxiliary head to predict a compact representation of the long-term future, preserving information relevant for long-form generations. We explore two variants of FSP: handcrafted summaries, for example, a bag of words summary of the future sequence, and learned summaries, which use embeddings produced by a reverse language model trained from right-to-left order. Large-scale pretraining experiments (3B and 8B-parameter models) demonstrate that FSP provides improvements over both NTP and MTP across math, reasoning, and coding benchmarks.
Cite
@article{arxiv.2510.14751,
title = {Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries},
author = {Divyat Mahajan and Sachin Goyal and Badr Youbi Idrissi and Mohammad Pezeshki and Ioannis Mitliagkas and David Lopez-Paz and Kartik Ahuja},
journal= {arXiv preprint arXiv:2510.14751},
year = {2026}
}
Comments
Proceedings of the Fourteenth International Conference on Learning Representations (ICLR) 2026