通过解耦优势策略优化实现闪电式思维
摘要
近期的大型推理模型 (LRM) 已通过监督微调 (SFT) 和强化学习 (RL) 在解决复杂问题方面取得了显著成绩。尽管现有的 RL 算法显著提高了模型准确性，但它们仍 suffer from excessively lengthy responses and overthinking issues， resulting in increased inference latency and computational consumption， especially for simple tasks that require minimal reasoning. To address this, we propose a novel RL framework, DEPO, to reduce inefficient reasoning for models. Our method mainly consists of three core components: (1) an innovative advantage decoupled algorithm to guide model reduction of inefficient tokens; (2) a difficulty-aware length penalty to lower the overall length of model responses; (3) an advantage clipping method to prevent bias in policy optimization. In our experiments, applied to DeepSeek-Distill-Qwen-7B and DeepSeek-Distill-Qwen-1.5B as base models, DEPO achieves a significant reduction in sequence length by 39% and reduces excessive reasoning paths in inefficient tokens, while outperforming the base model in overall accuracy.
引用
@article{arxiv.2510.15374,
title = {Towards Flash Thinking via Decoupled Advantage Policy Optimization},
author = {Zezhong Tan and Hang Gao and Xinhong Ma and Feng Zhang and Ziqiang Dong},
journal= {arXiv preprint arXiv:2510.15374},
year = {2025}
}