English

Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay

Computer Vision and Pattern Recognition 2021-06-21 v1 Machine Learning Neural and Evolutionary Computing

Abstract

This paper proposes two novel knowledge transfer techniques for class-incremental learning (CIL). First, we propose data-free generative replay (DF-GR) to mitigate catastrophic forgetting in CIL by using synthetic samples from a generative model. In the conventional generative replay, the generative model is pre-trained for old data and shared in extra memory for later incremental learning. In our proposed DF-GR, we train a generative model from scratch without using any training data, based on the pre-trained classification model from the past, so we curtail the cost of sharing pre-trained generative models. Second, we introduce dual-teacher information distillation (DT-ID) for knowledge distillation from two teachers to one student. In CIL, we use DT-ID to learn new classes incrementally based on the pre-trained model for old classes and another model (pre-)trained on the new data for new classes. We implemented the proposed schemes on top of one of the state-of-the-art CIL methods and showed the performance improvement on CIFAR-100 and ImageNet datasets.

Keywords

Cite

@article{arxiv.2106.09835,
  title  = {Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay},
  author = {Yoojin Choi and Mostafa El-Khamy and Jungwon Lee},
  journal= {arXiv preprint arXiv:2106.09835},
  year   = {2021}
}

Comments

CVPR 2021 Workshop on Continual Learning in Computer Vision (CLVision)

R2 v1 2026-06-24T03:20:24.464Z