In this paper, we propose the CodeRetriever model, which learns the function-level code semantic representations through large-scale code-text contrastive pre-training. We adopt two contrastive learning schemes in CodeRetriever: unimodal contrastive learning and bimodal contrastive learning. For unimodal contrastive learning, we design an unsupervised learning approach to build semantic-related code pairs based on the documentation and function name. For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build code-text pairs. Both contrastive objectives can fully leverage large-scale code corpus for pre-training. Extensive experimental results show that CodeRetriever achieves new state-of-the-art with significant improvement over existing code pre-trained models, on eleven domain/language-specific code search tasks with six programming languages in different code granularity (function-level, snippet-level and statement-level). These results demonstrate the effectiveness and robustness of CodeRetriever.
@article{arxiv.2201.10866,
title = {CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search},
author = {Xiaonan Li and Yeyun Gong and Yelong Shen and Xipeng Qiu and Hang Zhang and Bolun Yao and Weizhen Qi and Daxin Jiang and Weizhu Chen and Nan Duan},
journal= {arXiv preprint arXiv:2201.10866},
year = {2022}
}