Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet
Abstract
Recent years have seen a proliferation of attention mechanisms and the rise of Transformers in Natural Language Generation (NLG). Previously, state-of-the-art NLG architectures such as RNN and LSTM ran into vanishing gradient problems; as sentences grew larger, distance between positions remained linear, and sequential computation hindered parallelization since sentences were processed word by word. Transformers usher in a new era. In this paper, we explore three major Transformer-based models, namely GPT, BERT, and XLNet, that carry significant implications for the field. NLG is a burgeoning area that is now bolstered with rapid developments in attention mechanisms. From poetry generation to summarization, text generation derives benefit as Transformer-based language models achieve groundbreaking results.
Cite
@article{arxiv.2102.08036,
title = {Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet},
author = {M. Onat Topal and Anil Bas and Imke van Heerden},
journal= {arXiv preprint arXiv:2102.08036},
year = {2021}
}
Comments
Accepted as oral presentation to ICIDAAI 2021 - Short Paper