Finnish Language Modeling with Deep Transformer Models
Computation and Language
2020-03-30 v2 Machine Learning
Sound
Audio and Speech Processing
Machine Learning
Abstract
Transformers have recently taken the center stage in language modeling after LSTM's were considered the dominant model architecture for a long time. In this project, we investigate the performance of the Transformer architectures-BERT and Transformer-XL for the language modeling task. We use a sub-word model setting with the Finnish language and compare it to the previous State of the art (SOTA) LSTM model. BERT achieves a pseudo-perplexity score of 14.5, which is the first such measure achieved as far as we know. Transformer-XL improves upon the perplexity score to 73.58 which is 27\% better than the LSTM model.
Cite
@article{arxiv.2003.11562,
title = {Finnish Language Modeling with Deep Transformer Models},
author = {Abhilash Jain and Aku Ruohe and Stig-Arne Grönroos and Mikko Kurimo},
journal= {arXiv preprint arXiv:2003.11562},
year = {2020}
}
Comments
4 pages