Large language models have demonstrated strong reasoning capabilities in complex tasks through tool integration, which is typically framed as a Markov Decision Process and optimized with trajectory-level RL algorithms such as GRPO. However, a common class of reasoning tasks, iterative optimization, presents distinct challenges: the agent interacts with the same underlying environment state across turns, and the value of a trajectory is determined by the best turn-level reward rather than cumulative returns. Existing GRPO-based methods cannot perform fine-grained, turn-level optimization in such settings, while black-box optimization methods discard prior knowledge and reasoning capabilities. To address this gap, we propose Turn-Level GRPO (TL-GRPO), a lightweight RL algorithm that performs turn-level group sampling for fine-grained optimization. We evaluate TL-GRPO on analog circuit sizing (ACS), a challenging scientific optimization task requiring multiple simulations and domain expertise. Results show that TL-GRPO outperforms standard GRPO and Bayesian optimization methods across various specifications. Furthermore, our 30B model trained with TL-GRPO achieves state-of-the-art performance on ACS tasks under same simulation budget, demonstrating both strong generalization and practical utility.
@article{arxiv.2601.16480,
title = {TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimization},
author = {Peiji Li and Linyang Li and Handa Sun and Wenjin Mai and Yongkang Chen and Xiaozhe Li and Yue Shen and Yichuan Ma and Yiliu Sun and Jiaxi Cao and Zhishu He and Bo Wang and Xiaoqing Zheng and Zhaori Bi and Xipeng Qiu and Qipeng Guo and Kai Chen and Dahua Lin},
journal= {arXiv preprint arXiv:2601.16480},
year = {2026}
}