Distillation of encoder-decoder transformers for sequence labelling
Abstract
Driven by encouraging results on a wide range of tasks, the field of NLP is experiencing an accelerated race to develop bigger language models. This race for bigger models has also underscored the need to continue the pursuit of practical distillation approaches that can leverage the knowledge acquired by these big models in a compute-efficient manner. Having this goal in mind, we build on recent work to propose a hallucination-free framework for sequence tagging that is especially suited for distillation. We show empirical results of new state-of-the-art performance across multiple sequence labelling datasets and validate the usefulness of this framework for distilling a large model in a few-shot learning scenario.
Cite
@article{arxiv.2302.05454,
title = {Distillation of encoder-decoder transformers for sequence labelling},
author = {Marco Farina and Duccio Pappadopulo and Anant Gupta and Leslie Huang and Ozan İrsoy and Thamar Solorio},
journal= {arXiv preprint arXiv:2302.05454},
year = {2023}
}
Comments
Accepted to Findings of EACL 2023