In this work, we construct the largest dataset for multimodal pretraining in Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide range of domains. We propose a cross-modal pretraining method called M6, referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. We scale the model size up to 10 billion and 100 billion parameters, and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding performance in comparison with strong baselines. Furthermore, we specifically design a downstream task of text-guided image generation, and show that the finetuned M6 can create high-quality images with high resolution and abundant details.
@article{arxiv.2103.00823,
title = {M6: A Chinese Multimodal Pretrainer},
author = {Junyang Lin and Rui Men and An Yang and Chang Zhou and Ming Ding and Yichang Zhang and Peng Wang and Ang Wang and Le Jiang and Xianyan Jia and Jie Zhang and Jianwei Zhang and Xu Zou and Zhikang Li and Xiaodong Deng and Jie Liu and Jinbao Xue and Huiling Zhou and Jianxin Ma and Jin Yu and Yong Li and Wei Lin and Jingren Zhou and Jie Tang and Hongxia Yang},
journal= {arXiv preprint arXiv:2103.00823},
year = {2021}
}
Comments
12 pages, technical report. Extension of paper "M6" accepted to KDD 2021