Evaluation of OpenAI o1: Opportunities and Challenges of AGI
Abstract
This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer science, mathematics, natural sciences, medicine, linguistics, and social sciences. Through rigorous testing, o1-preview demonstrated remarkable capabilities, often achieving human-level or superior performance in areas ranging from coding challenges to scientific reasoning and from language processing to creative problem-solving. Key findings include: -83.3% success rate in solving complex competitive programming problems, surpassing many human experts. -Superior ability in generating coherent and accurate radiology reports, outperforming other evaluated models. -100% accuracy in high school-level mathematical reasoning tasks, providing detailed step-by-step solutions. -Advanced natural language inference capabilities across general and specialized domains like medicine. -Impressive performance in chip design tasks, outperforming specialized models in areas such as EDA script generation and bug analysis. -Remarkable proficiency in anthropology and geology, demonstrating deep understanding and reasoning in these specialized fields. -Strong capabilities in quantitative investing. O1 has comprehensive financial knowledge and statistical modeling skills. -Effective performance in social media analysis, including sentiment analysis and emotion recognition. The model excelled particularly in tasks requiring intricate reasoning and knowledge integration across various fields. While some limitations were observed, including occasional errors on simpler problems and challenges with certain highly specialized concepts, the overall results indicate significant progress towards artificial general intelligence.
Keywords
Cite
@article{arxiv.2409.18486,
title = {Evaluation of OpenAI o1: Opportunities and Challenges of AGI},
author = {Tianyang Zhong and Zhengliang Liu and Yi Pan and Yutong Zhang and Zeyu Zhang and Yifan Zhou and Shizhe Liang and Zihao Wu and Yanjun Lyu and Peng Shu and Xiaowei Yu and Chao Cao and Hanqi Jiang and Hanxu Chen and Yiwei Li and Junhao Chen and Huawen Hu and Yiheng Liu and Huaqin Zhao and Shaochen Xu and Haixing Dai and Lin Zhao and Ruidong Zhang and Wei Zhao and Zhenyuan Yang and Jingyuan Chen and Peilong Wang and Wei Ruan and Hui Wang and Huan Zhao and Jing Zhang and Yiming Ren and Shihuan Qin and Tong Chen and Jiaxi Li and Arif Hassan Zidan and Afrar Jahin and Minheng Chen and Sichen Xia and Jason Holmes and Yan Zhuang and Jiaqi Wang and Bochen Xu and Weiran Xia and Jichao Yu and Kaibo Tang and Yaxuan Yang and Bolun Sun and Tao Yang and Guoyu Lu and Xianqiao Wang and Lilong Chai and He Li and Jin Lu and Xin Zhang and Bao Ge and Xintao Hu and Lian Zhang and Hua Zhou and Lu Zhang and Shu Zhang and Zhen Xiang and Yudan Ren and Jun Liu and Xi Jiang and Yu Bao and Wei Zhang and Xiang Li and Gang Li and Wei Liu and Dinggang Shen and Andrea Sikora and Xiaoming Zhai and Dajiang Zhu and Tuo Zhang and Tianming Liu},
journal= {arXiv preprint arXiv:2409.18486},
year = {2025}
}