大模型路线图
机器学习
2022-04-21 v4 人工智能
计算与语言
摘要
随着深度学习的快速发展，为多个下游任务训练大模型（Big Models, BMs）成为一种流行范式。研究者已在 BM 的构建及 BM 在诸多领域的应用中取得各类成果。目前缺乏梳理 BM 整体进展并指导后续研究的工作。本文不仅涵盖 BM 技术本身，还包括 BM 训练与应用的前提条件，将 BM 综述分为四部分：资源、模型、关键技术与应用。我们在这四部分中介绍了 16 个具体的 BM 相关主题，分别为数据、知识、计算系统、并行训练系统、语言模型、视觉模型、多模态模型、理论与可解释性、常识推理、可靠性与安全、治理、评测、机器翻译、文本生成、对话与蛋白质研究。在每个主题中，我们清晰总结了当前研究并提出了若干未来研究方向。在本文末尾，我们以更通用的视角总结了 BM 的进一步发展。
引用
@article{arxiv.2203.14101,
title = {A Roadmap for Big Model},
author = {Sha Yuan and Hanyu Zhao and Shuai Zhao and Jiahong Leng and Yangxiao Liang and Xiaozhi Wang and Jifan Yu and Xin Lv and Zhou Shao and Jiaao He and Yankai Lin and Xu Han and Zhenghao Liu and Ning Ding and Yongming Rao and Yizhao Gao and Liang Zhang and Ming Ding and Cong Fang and Yisen Wang and Mingsheng Long and Jing Zhang and Yinpeng Dong and Tianyu Pang and Peng Cui and Lingxiao Huang and Zheng Liang and Huawei Shen and Hui Zhang and Quanshi Zhang and Qingxiu Dong and Zhixing Tan and Mingxuan Wang and Shuo Wang and Long Zhou and Haoran Li and Junwei Bao and Yingwei Pan and Weinan Zhang and Zhou Yu and Rui Yan and Chence Shi and Minghao Xu and Zuobai Zhang and Guoqiang Wang and Xiang Pan and Mengjie Li and Xiaoyu Chu and Zijun Yao and Fangwei Zhu and Shulin Cao and Weicheng Xue and Zixuan Ma and Zhengyan Zhang and Shengding Hu and Yujia Qin and Chaojun Xiao and Zheni Zeng and Ganqu Cui and Weize Chen and Weilin Zhao and Yuan Yao and Peng Li and Wenzhao Zheng and Wenliang Zhao and Ziyi Wang and Borui Zhang and Nanyi Fei and Anwen Hu and Zenan Ling and Haoyang Li and Boxi Cao and Xianpei Han and Weidong Zhan and Baobao Chang and Hao Sun and Jiawen Deng and Chujie Zheng and Juanzi Li and Lei Hou and Xigang Cao and Jidong Zhai and Zhiyuan Liu and Maosong Sun and Jiwen Lu and Zhiwu Lu and Qin Jin and Ruihua Song and Ji-Rong Wen and Zhouchen Lin and Liwei Wang and Hang Su and Jun Zhu and Zhifang Sui and Jiajun Zhang and Yang Liu and Xiaodong He and Minlie Huang and Jian Tang and Jie Tang},
journal= {arXiv preprint arXiv:2203.14101},
year = {2022}
}
备注
This report has been withdrawn by the authors due to critical issues in Section 2.3.1 of Article 2