Teacher-Class Network: A Neural Network Compression Mechanism
Abstract
To reduce the overwhelming size of Deep Neural Networks (DNN) teacher-student methodology tries to transfer knowledge from a complex teacher network to a simple student network. We instead propose a novel method called the teacher-class network consisting of a single teacher and multiple student networks (i.e. class of students). Instead of transferring knowledge to one student only, the proposed method transfers a chunk of knowledge to each student. Our students are not trained for problem-specific logits, they are trained to mimic knowledge (dense representation) learned by the teacher network thus the combined knowledge learned by the class of students can be used to solve other problems as well. The proposed teacher-class architecture is evaluated on several benchmark datasets such as MNIST, Fashion MNIST, IMDB Movie Reviews, CAMVid, CIFAR-10 and ImageNet on multiple tasks including image classification, sentiment classification and segmentation. Our approach outperforms the state of-the-art single student approach in terms of accuracy as well as computational cost while achieving 10-30 times reduction in parameters.
Cite
@article{arxiv.2004.03281,
title = {Teacher-Class Network: A Neural Network Compression Mechanism},
author = {Shaiq Munir Malik and Muhammad Umair Haider and Mohbat Tharani and Musab Rasheed and Murtaza Taj},
journal= {arXiv preprint arXiv:2004.03281},
year = {2021}
}
Comments
Published in BMVC 2021