Large language models (LLMs) integrated into multistep agent systems enable complex decision-making processes across various applications. However, their outputs often lack reliability, making uncertainty estimation crucial. Existing uncertainty estimation methods primarily focus on final-step outputs, which fail to account for cumulative uncertainty over the multistep decision-making process and the dynamic interactions between agents and their environments. To address these limitations, we propose SAUP (Situation Awareness Uncertainty Propagation), a novel framework that propagates uncertainty through each step of an LLM-based agent's reasoning process. SAUP incorporates situational awareness by assigning situational weights to each step's uncertainty during the propagation. Our method, compatible with various one-step uncertainty estimation techniques, provides a comprehensive and accurate uncertainty measure. Extensive experiments on benchmark datasets demonstrate that SAUP significantly outperforms existing state-of-the-art methods, achieving up to 20% improvement in AUROC.
@article{arxiv.2412.01033,
title = {SAUP: Situation Awareness Uncertainty Propagation on LLM Agent},
author = {Qiwei Zhao and Xujiang Zhao and Yanchi Liu and Wei Cheng and Yiyou Sun and Mika Oishi and Takao Osaki and Katsushi Matsuda and Huaxiu Yao and Haifeng Chen},
journal= {arXiv preprint arXiv:2412.01033},
year = {2024}
}