ANNA: Enhanced Language Representation for Question Answering
Abstract
Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance individually, and that the language model performs the best results on a specific question answering task when those approaches are jointly considered in pre-training models. In particular, we propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7\% F1 and 90.6\% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet on the SQuAD 2.0 benchmark.
Cite
@article{arxiv.2203.14507,
title = {ANNA: Enhanced Language Representation for Question Answering},
author = {Changwook Jun and Hansol Jang and Myoseop Sim and Hyun Kim and Jooyoung Choi and Kyungkoo Min and Kyunghoon Bae},
journal= {arXiv preprint arXiv:2203.14507},
year = {2022}
}
Comments
11 pages, 3 figures