The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling such large models to compact models for efficient deployment also necessitates a large amount of (labeled or unlabeled) training data. In this paper, we propose the teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through a large volume of data. TGT exploits the fact that the teacher has acquired a good representation of the underlying data domain, which typically corresponds to a much lower dimensional manifold than the input space. Furthermore, we can use the teacher to explore input space more efficiently through sampling or gradient-based methods; thus, making TGT especially attractive for limited data or long-tail settings. We formally capture this benefit of proposed data-domain exploration in our generalization bounds. We find that TGT can improve accuracy on several image classification benchmarks as well as a range of text classification and retrieval tasks.
@article{arxiv.2208.06825,
title = {Teacher Guided Training: An Efficient Framework for Knowledge Transfer},
author = {Manzil Zaheer and Ankit Singh Rawat and Seungyeon Kim and Chong You and Himanshu Jain and Andreas Veit and Rob Fergus and Sanjiv Kumar},
journal= {arXiv preprint arXiv:2208.06825},
year = {2022}
}