What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
Abstract
GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.
Cite
@article{arxiv.2109.04650,
title = {What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers},
author = {Boseop Kim and HyoungSeok Kim and Sang-Woo Lee and Gichang Lee and Donghyun Kwak and Dong Hyeon Jeon and Sunghyun Park and Sungju Kim and Seonhoon Kim and Dongpil Seo and Heungsub Lee and Minyoung Jeong and Sungjae Lee and Minsub Kim and Suk Hyun Ko and Seokhun Kim and Taeyong Park and Jinuk Kim and Soyoung Kang and Na-Hyeon Ryu and Kang Min Yoo and Minsuk Chang and Soobin Suh and Sookyo In and Jinseong Park and Kyungduk Kim and Hiun Kim and Jisu Jeong and Yong Goo Yeo and Donghoon Ham and Dongju Park and Min Young Lee and Jaewook Kang and Inho Kang and Jung-Woo Ha and Woomyoung Park and Nako Sung},
journal= {arXiv preprint arXiv:2109.04650},
year = {2021}
}
Comments
Accepted to EMNLP2021 as a long paper. Fixed some typos