We introduce a novel approach to transformers that learns hierarchical representations in multiparty dialogue. First, three language modeling tasks are used to pre-train the transformers, token- and utterance-level language modeling and utterance order prediction, that learn both token and utterance embeddings for better understanding in dialogue contexts. Then, multi-task learning between the utterance prediction and the token span prediction is applied to fine-tune for span-based question answering (QA). Our approach is evaluated on the FriendsQA dataset and shows improvements of 3.8% and 1.4% over the two state-of-the-art transformer models, BERT and RoBERTa, respectively.
@article{arxiv.2004.03561,
title = {Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering},
author = {Changmao Li and Jinho D. Choi},
journal= {arXiv preprint arXiv:2004.03561},
year = {2020}
}
Comments
Accepted by the Annual Conference of the Association for Computational Linguistics, ACL 2020