基于科学家对齐工作流程的LLM科学通用智能探针
人工智能
2025-12-22 v1 计算与语言
机器学习
摘要
尽管科学AI取得了进展，但尚无统一框架来衡量科学通用智能（SGI）——即能够自主构思、探究和跨学科推理的能力。我们提出一种以Practical Inquiry Model（PIM：Deliberation, Conception, Action, Perception）为基础的SGI定义，并通过四个科学家对齐任务来实现其操作化：深层研究、想象生成、干/湿实验和实验推理。SGI-Bench包含超过1,000个由专家精选、跨学科的样本，灵感来自Science的125大问题，使对最新水平LLM进行系统评估成为可能。结果揭示了差距：在深层研究中尽管存在步骤级对齐，却出现低于10-20%的精确匹配率；想象缺乏可行性和细节；干实验中代码可执行性高但执行结果准确性低；湿实验程序序列保真度低；持续存在多模态比较推理挑战。我们进一步引入Test-Time Reinforcement Learning（TTRL），通过优化检索增强的新颖性奖励于推理阶段，提升假设的新颖性而无需参考答案。总之，我们的PIM基础定义、工作流程中心的基准以及实证见解，为能够真正参与科学发现的AI系统奠定了基础。
关键词
引用
@article{arxiv.2512.16969,
title = {Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows},
author = {Wanghan Xu and Yuhao Zhou and Yifan Zhou and Qinglong Cao and Shuo Li and Jia Bu and Bo Liu and Yixin Chen and Xuming He and Xiangyu Zhao and Xiang Zhuang and Fengxiang Wang and Zhiwang Zhou and Qiantai Feng and Wenxuan Huang and Jiaqi Wei and Hao Wu and Yuejin Yang and Guangshuai Wang and Sheng Xu and Ziyan Huang and Xinyao Liu and Jiyao Liu and Cheng Tang and Wei Li and Ying Chen and Junzhi Ning and Pengfei Jiang and Chenglong Ma and Ye Du and Changkai Ji and Huihui Xu and Ming Hu and Jiangbin Zheng and Xin Chen and Yucheng Wu and Feifei Jiang and Xi Chen and Xiangru Tang and Yuchen Fu and Yingzhou Lu and Yuanyuan Zhang and Lihao Sun and Chengbo Li and Jinzhe Ma and Wanhao Liu and Yating Liu and Kuo-Cheng Wu and Shengdu Chai and Yizhou Wang and Ouwen Zhangjin and Chen Tang and Shufei Zhang and Wenbo Cao and Junjie Ren and Taoyong Cui and Zhouheng Yao and Juntao Deng and Yijie Sun and Feng Liu and Wangxu Wei and Jingyi Xu and Zhangrui Li and Junchao Gong and Zijie Guo and Zhiyu Yao and Zaoyu Chen and Tianhao Peng and Fangchen Yu and Bo Zhang and Dongzhan Zhou and Shixiang Tang and Jiaheng Liu and Fenghua Ling and Yan Lu and Yuchen Ren and Ben Fei and Zhen Zhao and Xinyu Gu and Rui Su and Xiao-Ming Wu and Weikang Si and Yang Liu and Hao Chen and Xiangchao Yan and Xue Yang and Junchi Yan and Jiamin Wu and Qihao Zheng and Chenhui Li and Zhiqiang Gao and Hao Kong and Junjun He and Mao Su and Tianfan Fu and Peng Ye and Chunfeng Song and Nanqing Dong and Yuqiang Li and Huazhu Fu and Siqi Sun and Lijing Cheng and Jintai Lin and Wanli Ouyang and Bowen Zhou and Wenlong Zhang and Lei Bai},
journal= {arXiv preprint arXiv:2512.16969},
year = {2025}
}