As the capabilities of code large language models (LLMs) continue to expand, their applications across diverse code intelligence domains are rapidly increasing. However, most existing datasets only evaluate limited application domains. To address this gap, we have developed a comprehensive code evaluation dataset FullStack Bench focusing on full-stack programming, which encompasses a wide range of application domains (e.g., basic programming, data analysis, software engineering, mathematics, and machine learning). Besides, to assess multilingual programming capabilities, in FullStack Bench, we design real-world instructions and corresponding unit test cases from 16 widely-used programming languages to reflect real-world usage scenarios rather than simple translations. Moreover, we also release an effective code sandbox execution tool (i.e., SandboxFusion) supporting various programming languages and packages to evaluate the performance of our FullStack Bench efficiently. Comprehensive experimental results on our FullStack Bench demonstrate the necessity and effectiveness of our FullStack Bench and SandboxFusion.
@article{arxiv.2412.00535,
title = {FullStack Bench: Evaluating LLMs as Full Stack Coders},
author = {Bytedance-Seed-Foundation-Code-Team and : and Yao Cheng and Jianfeng Chen and Jie Chen and Li Chen and Liyu Chen and Wentao Chen and Zhengyu Chen and Shijie Geng and Aoyan Li and Bo Li and Bowen Li and Linyi Li and Boyi Liu and Jiaheng Liu and Kaibo Liu and Qi Liu and Shukai Liu and Siyao Liu and Tianyi Liu and Tingkai Liu and Yongfei Liu and Rui Long and Jing Mai and Guanghan Ning and Z. Y. Peng and Kai Shen and Jiahao Su and Jing Su and Tao Sun and Yifan Sun and Yunzhe Tao and Guoyin Wang and Siwei Wang and Xuwu Wang and Yite Wang and Zihan Wang and Jinxiang Xia and Liang Xiang and Xia Xiao and Yongsheng Xiao and Chenguang Xi and Shulin Xin and Jingjing Xu and Shikun Xu and Hongxia Yang and Jack Yang and Yingxiang Yang and Jianbo Yuan and Jun Zhang and Yufeng Zhang and Yuyu Zhang and Shen Zheng and He Zhu and Ming Zhu},
journal= {arXiv preprint arXiv:2412.00535},
year = {2025}
}