Large-scale vision-language models (VLMs) have shown a strong zero-shot generalization capability on unseen-domain data. However, adapting pre-trained VLMs to a sequence of downstream tasks often leads to the forgetting of previously learned knowledge and a reduction in zero-shot classification performance. To tackle this problem, we propose a unique Selective Dual-Teacher Knowledge Transfer framework that leverages the most recent fine-tuned and the original pre-trained VLMs as dual teachers to preserve the previously learned knowledge and zero-shot capabilities, respectively. With only access to an unlabeled reference dataset, our proposed framework performs a selective knowledge distillation mechanism by measuring the feature discrepancy from the dual-teacher VLMs. Consequently, our selective dual-teacher knowledge distillation mitigates catastrophic forgetting of previously learned knowledge while preserving the zero-shot capabilities of pre-trained VLMs. Extensive experiments on benchmark datasets demonstrate that our framework is favorable against state-of-the-art continual learning approaches for preventing catastrophic forgetting and zero-shot degradation. Project page: https://chuyu.org/research/snd
@article{arxiv.2403.09296,
title = {Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models},
author = {Yu-Chu Yu and Chi-Pin Huang and Jr-Jen Chen and Kai-Po Chang and Yung-Hsuan Lai and Fu-En Yang and Yu-Chiang Frank Wang},
journal= {arXiv preprint arXiv:2403.09296},
year = {2024}
}
Comments
Accepted to ECCV 2024. Project page: https://chuyu.org/research/snd