Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
Abstract
Despite advances in scientific AI, a coherent framework for Scientific General Intelligence (SGI)-the ability to autonomously conceive, investigate, and reason across scientific domains-remains lacking. We present an operational SGI definition grounded in the Practical Inquiry Model (PIM: Deliberation, Conception, Action, Perception) and operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning. SGI-Bench comprises over 1,000 expert-curated, cross-disciplinary samples inspired by Science's 125 Big Questions, enabling systematic evaluation of state-of-the-art LLMs. Results reveal gaps: low exact match (10--20%) in deep research despite step-level alignment; ideas lacking feasibility and detail; high code executability but low execution result accuracy in dry experiments; low sequence fidelity in wet protocols; and persistent multimodal comparative-reasoning challenges. We further introduce Test-Time Reinforcement Learning (TTRL), which optimizes retrieval-augmented novelty rewards at inference, enhancing hypothesis novelty without reference answer. Together, our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.
Cite
@article{arxiv.2512.16969,
title = {Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows},
author = {Wanghan Xu and Yuhao Zhou and Yifan Zhou and Qinglong Cao and Shuo Li and Jia Bu and Bo Liu and Yixin Chen and Xuming He and Xiangyu Zhao and Xiang Zhuang and Fengxiang Wang and Zhiwang Zhou and Qiantai Feng and Wenxuan Huang and Jiaqi Wei and Hao Wu and Yuejin Yang and Guangshuai Wang and Sheng Xu and Ziyan Huang and Xinyao Liu and Jiyao Liu and Cheng Tang and Wei Li and Ying Chen and Junzhi Ning and Pengfei Jiang and Chenglong Ma and Ye Du and Changkai Ji and Huihui Xu and Ming Hu and Jiangbin Zheng and Xin Chen and Yucheng Wu and Feifei Jiang and Xi Chen and Xiangru Tang and Yuchen Fu and Yingzhou Lu and Yuanyuan Zhang and Lihao Sun and Chengbo Li and Jinzhe Ma and Wanhao Liu and Yating Liu and Kuo-Cheng Wu and Shengdu Chai and Yizhou Wang and Ouwen Zhangjin and Chen Tang and Shufei Zhang and Wenbo Cao and Junjie Ren and Taoyong Cui and Zhouheng Yao and Juntao Deng and Yijie Sun and Feng Liu and Wangxu Wei and Jingyi Xu and Zhangrui Li and Junchao Gong and Zijie Guo and Zhiyu Yao and Zaoyu Chen and Tianhao Peng and Fangchen Yu and Bo Zhang and Dongzhan Zhou and Shixiang Tang and Jiaheng Liu and Fenghua Ling and Yan Lu and Yuchen Ren and Ben Fei and Zhen Zhao and Xinyu Gu and Rui Su and Xiao-Ming Wu and Weikang Si and Yang Liu and Hao Chen and Xiangchao Yan and Xue Yang and Junchi Yan and Jiamin Wu and Qihao Zheng and Chenhui Li and Zhiqiang Gao and Hao Kong and Junjun He and Mao Su and Tianfan Fu and Peng Ye and Chunfeng Song and Nanqing Dong and Yuqiang Li and Huazhu Fu and Siqi Sun and Lijing Cheng and Jintai Lin and Wanli Ouyang and Bowen Zhou and Wenlong Zhang and Lei Bai},
journal= {arXiv preprint arXiv:2512.16969},
year = {2025}
}