Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modeling
Abstract
In Agentic Search, trajectory-level outcome rewards fail to quantify the behavioral contributions of individual steps, while existing step-level reward methods typically rely on costly tree sampling. We view world knowledge as a latent world graph and each IS task as search within a latent task graph, where effective steps should make graph progress toward the answer node. Based on this prior, we propose Graph-Distance Contribution Reward (GDCR), a step-level process reward that scores newly-retrieved and newly-cited entities by their distance to the answer node in a training-time Entity-Relation (ER) graph. We further propose Step Advantage Policy Optimization (SAPO), which converts GDCR into step-level advantages and combines them with trajectory-level outcome advantages. Experiments on four challenging benchmarks validate the effectiveness of our method.
Comments: 15 pages, 8 figures
Cite
@article{arxiv.2605.29697,
title = {Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modeling},
author = {Yuchen Liu and Yingjie Feng and Lixiong Qin and Jiasi Chen and Jianing Yu and Sheng Gao and Sheng Yang and Weiran Xu},
journal= {arXiv preprint arXiv:2605.29697},
year = {2026}
}