Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

Yige Xu; Xipeng Qiu; Ligao Zhou; Xuanjing Huang

Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

Computation and Language 2020-02-25 v1 Machine Learning

Authors: Yige Xu , Xipeng Qiu , Ligao Zhou , Xuanjing Huang

Abstract

Fine-tuning pre-trained language models like BERT has become an effective way in NLP and yields state-of-the-art results on many downstream tasks. Recent studies on adapting BERT to new tasks mainly focus on modifying the model structure, re-designing the pre-train tasks, and leveraging external data and knowledge. The fine-tuning strategy itself has yet to be fully explored. In this paper, we improve the fine-tuning of BERT with two effective mechanisms: self-ensemble and self-distillation. The experiments on text classification and natural language inference tasks show our proposed methods can significantly improve the adaption of BERT without any external data or knowledge.

Keywords

bert pre-trained language model supervised fine-tuning

Cite

@article{arxiv.2002.10345,
  title  = {Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation},
  author = {Yige Xu and Xipeng Qiu and Ligao Zhou and Xuanjing Huang},
  journal= {arXiv preprint arXiv:2002.10345},
  year   = {2020}
}

Comments

7 pages, 6 figures

Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

Abstract

Keywords

Cite

Comments

Related papers