Czert -- Czech BERT-like Model for Language Representation
Computation and Language
2021-08-23 v3
Abstract
This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss properties of monolingual and multilingual models based upon our results. We publish all the pre-trained and fine-tuned models freely for the research community.
Cite
@article{arxiv.2103.13031,
title = {Czert -- Czech BERT-like Model for Language Representation},
author = {Jakub Sido and Ondřej Pražák and Pavel Přibáň and Jan Pašek and Michal Seják and Miloslav Konopík},
journal= {arXiv preprint arXiv:2103.13031},
year = {2021}
}
Comments
13 pages