Large language models (LLMs) deployed as agents solve user-specified tasks over multiple steps while keeping the required manual engagement to a minimum. Crucially, such LLMs need to ground their generations in any feedback obtained to reliably achieve the desired outcomes. We propose an end-to-end reinforcement learning method for teaching models to leverage execution feedback in the realm of code synthesis, where state-of-the-art LLMs struggle to improve code iteratively compared to independent sampling. We benchmark on competitive programming tasks, where we achieve new state-of-the art results with both small (8B parameters) and large (70B) models while reducing the amount of samples required by an order of magnitude. Our analysis of inference-time behavior demonstrates that our method produces LLMs that effectively leverage automatic feedback over multiple steps.
@article{arxiv.2410.02089,
title = {RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning},
author = {Jonas Gehring and Kunhao Zheng and Jade Copet and Vegard Mella and Quentin Carbonneaux and Taco Cohen and Gabriel Synnaeve},
journal= {arXiv preprint arXiv:2410.02089},
year = {2025}
}