LLM-Cave：面向大语言模型推理与决策系统的基准与轻量环境
天体物理仪器与方法
2025-12-01 v1 高能天体物理现象
广义相对论与量子宇宙学
摘要
大型语言模型（LLM）如 ChatGPT o1、ChatGPT o3 和 DeepSeek R1 在解决复杂问题方面展现出巨大潜力。然而，当前的 LLM 评估基准仅限于单步交互；部分现有序列决策环境（如 TextStarCraftII 和 LLM-PySC2）过于复杂，需数小时才能完成游戏。在本文中，我们引入 LLM-Cave，一个用于 LLM 推理与决策系统的基准与轻量环境。该环境是符号主义时代的经典实例，人工智能代理可通过推理附近危险并利用部分可观测状态信息来探索环境并规避潜在损失。在实验中，我们评估了主流大型语言模型（如 GPT-4o-mini、o1-mini 和 DeepSeek-R1）的顺序推理能力、决策性能和计算效率。实验表明，尽管 DeepSeek-R1 在复杂推理任务中实现了最高成功率，但更小的模型如 4o-mini 通过采用链式推演（Chain of Speculation）和计划-批判（Planner-Critic）策略显著缩小了在挑战性任务中的性能差距，同时以牺牲计算效率为代价。这表明，结构化的多步骤推理结合 LLM 反馈机制可显著提升 LLM 的决策能力，为改进弱模型推理提供了可行方向，并为 LLM 评估提出了以推理为中心的新型基准。我们的代码已开源于 https://github.com/puleya1277/CaveEnv。
引用
@article{arxiv.2511.22597,
title = {The NANOGrav 12.5-year Data Set: Chromatic Noise Characterization & Mitigation with Time-Domain Kernels},
author = {Jeffrey S. Hazboun and Joseph Simon and Jeremy Baier and Bjorn Larsen and Daniel J. Oliver and Paul T. Baker and Bence Bécsy and Siyuan Chen and Alberto Diaz Hernandez and Justin A. Ellis and A. Miguel Holgado and Kristina Islo and Aaron Johnson and Andrew R. Kaiser and Nima Laal and Alexander McEwen and Nihan S. Pol and Joey Shapiro Key and Min Young Kim and Matthew Samson and Brent J. Shapiro-Albert and Jerry P. Sun and Stephen R. Taylor and Caitlin A. Witt and Jeremy Volpe and Christine Ye and Harsha Blumer and Paul R. Brook and Shami Chatterjee and James M. Cordes and Fronefield Crawford and H. Thankful Cromartie and Megan E. DeCesar and Paul B. Demorest and Timothy Dolch and Robert D. Ferdman and Elizabeth C. Ferrara and William Fiore and Emmanuel Fonseca and Nathan Garver-Daniels and Peter A. Gentile and Deborah C. Good and Ross J. Jennings and Megan L. Jones and David L. Kaplan and Michael T. Lam and T. Joseph W. Lazio and Duncan R. Lorimer and Jing Luo and Ryan S. Lynch and Dustin R. Madison and Maura A. McLaughlin and Chiara M. F. Mingarelli and Cherry Ng and David J. Nice and Timothy T. Pennucci and Scott M. Ransom and Paul S. Ray and Xavier Siemens and Renée Spiewak and Ingrid H. Stairs and Daniel R. Stinebring and Kevin Stovall and Joseph K. Swiggum and Jacob E. Turner and Michele Vallisneri and Sarah J. Vigeland},
journal= {arXiv preprint arXiv:2511.22597},
year = {2025}
}
备注
30 pages, 12 Figures, 6 Tables