LingBench++：一个用于大语言模型多步与跨文化推理的语言学信息基准与推理框架
计算与语言
2025-07-25 v2
摘要
我们提出了LingBench++，这是一个基于语言学信息的基准和推理框架，旨在评估大型语言模型(LLM)在受国际语言学奥林匹克竞赛(IOL)启发的复杂语言学任务上的表现。与仅关注最终答案准确性的先前基准不同，LingBench++提供了结构化的推理轨迹、逐步评估协议以及涵盖超过90种低资源和跨文化语言的丰富类型学元数据。我们进一步开发了一种多智能体架构，集成了语法知识检索、工具增强推理和审慎的假设检验。通过基线模型与我们提出的智能体模型的系统比较，我们证明，配备外部知识源和迭代推理的模型在准确性和可解释性方面均优于单次通过方法。LingBench++为推进LLM中基于语言学、具有文化意识且认知上合理的推理提供了全面的基础。
引用
@article{arxiv.2507.16809,
title = {LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs},
author = {Da-Chen Lian and Ri-Sheng Huang and Pin-Er Chen and Chunki Lim and You-Kuan Lin and Guan-Yu Tseng and Zi-Cheng Yang and Zhen-Yu Lin and Pin-Cheng Chen and Shu-Kai Hsieh},
journal= {arXiv preprint arXiv:2507.16809},
year = {2025}
}
备注
42p, 17f, 10t. Revisions: Merged paragraphs in Intro to emphasize contributions. Clarified benchmark design (Sec 3.5.1). Added single-agent, OpenAI-guided & 6-round experiments (Sec 5.2). Note: we only ran each experiment once; statistical tests are needed for strong claims. Revised Sec 6. Added acknowledgements, 2 new co-authors, and corrected typos/grammar