Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging: a single success-or-failure signal must be distributed across many actions. Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback. Multi-turn agent settings are underexplored, where feedback can include error messages, page changes, observations, or reference trajectories. We systematically study five feedback sources and two insertion granularities and introduce SERL, a selective environment-reweighted learning framework. SERL uses the task reward to determine update direction, while environment feedback adjusts placement and magnitude, focusing on critical actions. On ALFWorld and WebShop, SERL achieves 90.0% and 80.1% success, outperforming strong RL and distillation baselines. Analysis shows that grounded, action-relevant feedback at meaningful points consistently outperforms indiscriminate use of longer or richer context.
@article{arxiv.2605.19447,
title = {What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents},
author = {Xiaozhe Li and Tianyi Lyu and Yang Li and Yichuan Ma and Peiji Li and Linyang Li and Qipeng Guo and Dahua Lin and Kai Chen},
journal= {arXiv preprint arXiv:2605.19447},
year = {2026}
}