Large-scale pre-trained sequence-to-sequence models like BART and T5 achieve state-of-the-art performance on many generative NLP tasks. However, such models pose a great challenge in resource-constrained scenarios owing to their large memory requirements and high latency. To alleviate this issue, we propose to jointly distill and quantize the model, where knowledge is transferred from the full-precision teacher model to the quantized and distilled low-precision student model. Empirical analyses show that, despite the challenging nature of generative tasks, we were able to achieve a 16.5x model footprint compression ratio with little performance drop relative to the full-precision counterparts on multiple summarization and QA datasets. We further pushed the limit of compression ratio to 27.7x and presented the performance-efficiency trade-off for generative tasks using pre-trained models. To the best of our knowledge, this is the first work aiming to effectively distill and quantize sequence-to-sequence pre-trained models for language generation tasks.
@article{arxiv.2203.11239,
title = {DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization},
author = {Zheng Li and Zijian Wang and Ming Tan and Ramesh Nallapati and Parminder Bhatia and Andrew Arnold and Bing Xiang and Dan Roth},
journal= {arXiv preprint arXiv:2203.11239},
year = {2022}
}