A Roadmap for Big Model
Abstract
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm. Researchers have achieved various outcomes in the construction of BMs and the BM application in many fields. At present, there is a lack of research work that sorts out the overall progress of BMs and guides the follow-up research. In this paper, we cover not only the BM technologies themselves but also the prerequisites for BM training and applications with BMs, dividing the BM review into four parts: Resource, Models, Key Technologies and Application. We introduce 16 specific BM-related topics in those four parts, they are Data, Knowledge, Computing System, Parallel Training System, Language Model, Vision Model, Multi-modal Model, Theory&Interpretability, Commonsense Reasoning, Reliability&Security, Governance, Evaluation, Machine Translation, Text Generation, Dialogue and Protein Research. In each topic, we summarize clearly the current studies and propose some future research directions. At the end of this paper, we conclude the further development of BMs in a more general view.
Cite
@article{arxiv.2203.14101,
title = {A Roadmap for Big Model},
author = {Sha Yuan and Hanyu Zhao and Shuai Zhao and Jiahong Leng and Yangxiao Liang and Xiaozhi Wang and Jifan Yu and Xin Lv and Zhou Shao and Jiaao He and Yankai Lin and Xu Han and Zhenghao Liu and Ning Ding and Yongming Rao and Yizhao Gao and Liang Zhang and Ming Ding and Cong Fang and Yisen Wang and Mingsheng Long and Jing Zhang and Yinpeng Dong and Tianyu Pang and Peng Cui and Lingxiao Huang and Zheng Liang and Huawei Shen and Hui Zhang and Quanshi Zhang and Qingxiu Dong and Zhixing Tan and Mingxuan Wang and Shuo Wang and Long Zhou and Haoran Li and Junwei Bao and Yingwei Pan and Weinan Zhang and Zhou Yu and Rui Yan and Chence Shi and Minghao Xu and Zuobai Zhang and Guoqiang Wang and Xiang Pan and Mengjie Li and Xiaoyu Chu and Zijun Yao and Fangwei Zhu and Shulin Cao and Weicheng Xue and Zixuan Ma and Zhengyan Zhang and Shengding Hu and Yujia Qin and Chaojun Xiao and Zheni Zeng and Ganqu Cui and Weize Chen and Weilin Zhao and Yuan Yao and Peng Li and Wenzhao Zheng and Wenliang Zhao and Ziyi Wang and Borui Zhang and Nanyi Fei and Anwen Hu and Zenan Ling and Haoyang Li and Boxi Cao and Xianpei Han and Weidong Zhan and Baobao Chang and Hao Sun and Jiawen Deng and Chujie Zheng and Juanzi Li and Lei Hou and Xigang Cao and Jidong Zhai and Zhiyuan Liu and Maosong Sun and Jiwen Lu and Zhiwu Lu and Qin Jin and Ruihua Song and Ji-Rong Wen and Zhouchen Lin and Liwei Wang and Hang Su and Jun Zhu and Zhifang Sui and Jiajun Zhang and Yang Liu and Xiaodong He and Minlie Huang and Jian Tang and Jie Tang},
journal= {arXiv preprint arXiv:2203.14101},
year = {2022}
}
Comments
This report has been withdrawn by the authors due to critical issues in Section 2.3.1 of Article 2