Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey
Abstract
Deliberative tree search is a cornerstone of modern Large Language Model (LLM) research, driving the pivot from brute-force scaling toward algorithmic efficiency. This single paradigm unifies two critical frontiers: \textbf{Test-Time Scaling (TTS)}, which deploys on-demand computation to solve hard problems, and \textbf{Self-Improvement}, which uses search-generated data to durably enhance model parameters. However, this burgeoning field is fragmented and lacks a common formalism, particularly concerning the ambiguous role of the reward signal -- is it a transient heuristic or a durable learning target? This paper resolves this ambiguity by introducing a unified framework that deconstructs search algorithms into three core components: the \emph{Search Mechanism}, \emph{Reward Formulation}, and \emph{Transition Function}. We establish a formal distinction between transient \textbf{Search Guidance} for TTS and durable \textbf{Parametric Reward Modeling} for Self-Improvement. Building on this formalism, we introduce a component-centric taxonomy, synthesize the state-of-the-art, and chart a research roadmap toward more systematic progress in creating autonomous, self-improving agents.
Cite
@article{arxiv.2510.09988,
title = {Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey},
author = {Jiaqi Wei and Xiang Zhang and Yuejin Yang and Wenxuan Huang and Juntai Cao and Sheng Xu and Xiang Zhuang and Zhangyang Gao and Muhammad Abdul-Mageed and Laks V. S. Lakshmanan and Chenyu You and Wanli Ouyang and Siqi Sun},
journal= {arXiv preprint arXiv:2510.09988},
year = {2025}
}