We present PanGu-Coder, a pretrained decoder-only language model adopting the PanGu-Alpha architecture for text-to-code generation, i.e. the synthesis of programming language solutions given a natural language problem description. We train PanGu-Coder using a two-stage strategy: the first stage employs Causal Language Modelling (CLM) to pre-train on raw programming language data, while the second stage uses a combination of Causal Language Modelling and Masked Language Modelling (MLM) training objectives that focus on the downstream task of text-to-code generation and train on loosely curated pairs of natural language program definitions and code functions. Finally, we discuss PanGu-Coder-FT, which is fine-tuned on a combination of competitive programming problems and code with continuous integration tests. We evaluate PanGu-Coder with a focus on whether it generates functionally correct programs and demonstrate that it achieves equivalent or better performance than similarly sized models, such as CodeX, while attending a smaller context window and training on less data.
@article{arxiv.2207.11280,
title = {PanGu-Coder: Program Synthesis with Function-Level Language Modeling},
author = {Fenia Christopoulou and Gerasimos Lampouras and Milan Gritta and Guchun Zhang and Yinpeng Guo and Zhongqi Li and Qi Zhang and Meng Xiao and Bo Shen and Lin Li and Hao Yu and Li Yan and Pingyi Zhou and Xin Wang and Yuchi Ma and Ignacio Iacobacci and Yasheng Wang and Guangtai Liang and Jiansheng Wei and Xin Jiang and Qianxiang Wang and Qun Liu},
journal= {arXiv preprint arXiv:2207.11280},
year = {2022}
}