评估基于代码训练的大语言模型
机器学习
2021-07-15 v2
摘要
我们介绍了 Codex，一个在 GitHub 公开可用代码上微调的 GPT 语言模型，并研究其 Python 代码编写能力。Codex 的一个独立生产版本为 GitHub Copilot 提供动力。在 HumanEval（我们发布用于衡量从文档字符串合成程序功能正确性的新评估集）上，我们的模型解决了 28.8% 的问题，而 GPT-3 解决 0%，GPT-J 解决 11.4%。此外，我们发现从模型中重复采样对于生成困难提示的可行解决方案是一种出奇有效的策略。使用该方法，我们以每个问题 100 个样本解决了 70.2% 的问题。对我们模型的仔细调查揭示了其局限性，包括难以处理描述长操作链以及将操作绑定到变量的文档字符串。最后，我们讨论了部署强大代码生成技术的潜在广泛影响，涵盖安全性、保障性和经济学。
引用
@article{arxiv.2107.03374,
title = {Evaluating Large Language Models Trained on Code},
author = {Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Ponde de Oliveira Pinto and Jared Kaplan and Harri Edwards and Yuri Burda and Nicholas Joseph and Greg Brockman and Alex Ray and Raul Puri and Gretchen Krueger and Michael Petrov and Heidy Khlaaf and Girish Sastry and Pamela Mishkin and Brooke Chan and Scott Gray and Nick Ryder and Mikhail Pavlov and Alethea Power and Lukasz Kaiser and Mohammad Bavarian and Clemens Winter and Philippe Tillet and Felipe Petroski Such and Dave Cummings and Matthias Plappert and Fotios Chantzis and Elizabeth Barnes and Ariel Herbert-Voss and William Hebgen Guss and Alex Nichol and Alex Paino and Nikolas Tezak and Jie Tang and Igor Babuschkin and Suchir Balaji and Shantanu Jain and William Saunders and Christopher Hesse and Andrew N. Carr and Jan Leike and Josh Achiam and Vedant Misra and Evan Morikawa and Alec Radford and Matthew Knight and Miles Brundage and Mira Murati and Katie Mayer and Peter Welinder and Bob McGrew and Dario Amodei and Sam McCandlish and Ilya Sutskever and Wojciech Zaremba},
journal= {arXiv preprint arXiv:2107.03374},
year = {2021}
}
备注
corrected typos, added references, added authors, added acknowledgements