Large Language Models (LLMs) often produce answers with a single chain-of-thought, which restricts their ability to explore reasoning paths or self-correct flawed outputs in complex tasks. In this paper, we introduce MALT (Multi-Agent LLM Training), a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps using a sequential pipeline of heterogeneous agents. During data generation, each agent is repeatedly sampled to form a multi-agent search tree, where final outputs are graded against ground-truth data. We then apply value iteration to propagate reward signals back to each role-conditioned model, automatically producing multi-agent post-training data without human or teacher-model supervision. Our off-policy approach allows each agent to specialize by learning from correct and incorrect trajectories, ultimately improving the end-to-end reasoning chain. On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively, making it an important advance towards multi-agent cooperative training.
@article{arxiv.2412.01928,
title = {MALT: Improving Reasoning with Multi-Agent LLM Training},
author = {Sumeet Ramesh Motwani and Chandler Smith and Rocktim Jyoti Das and Rafael Rafailov and Ivan Laptev and Philip H. S. Torr and Fabio Pizzati and Ronald Clark and Christian Schroeder de Witt},
journal= {arXiv preprint arXiv:2412.01928},
year = {2025}
}