Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models

Yu-Chu Yu, Chi-Pin Huang, Jr-Jen Chen, Kai-Po Chang, Yung-Hsuan Lai +2 more
2/14/2026

No official code found. Best implementation: chu0802/SnD (17 stars, MIT, last commit 2025-09-24).

Abstract

Large-scale vision-language models (VLMs) have shown a strong zero-shot generalization capability on unseen-domain data. However, adapting pre-trained VLMs to a sequence of downstream tasks often leads to the forgetting of previously learned knowledge and a reduction in zero-shot classification performance. To tackle this problem, we propose a unique Selective Dual-Teacher Knowledge Transfer framework that leverages the most recent fine-tuned and the original pre-trained VLMs as dual teachers to preserve the previously learned knowledge and zero-shot capabilities, respectively. With only access to an unlabeled reference dataset, our proposed framework performs a selective knowledge distillation mechanism by measuring the feature discrepancy from the dual-teacher VLMs. Consequently, our selective dual-teacher knowledge distillation mitigates catastrophic forgetting of previously learned knowledge while preserving the zero-shot capabilities of pre-trained VLMs. Extensive experiments on benchmark datasets demonstrate that our framework is favorable against state-of-the-art continual learning approaches for preventing catastrophic forgetting and zero-shot degradation. Project page: https://chuyu.org/research/snd

View on arXivView PDF

Code Implementations(1)

chu0802/SnDCommunity62%
MIT

This is an official implementation of our work, Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models, accepted to ECCV'24

171PythonJul 1, 20241 years agoMIT
continual-learningeccveccv2024vision-language-models

Similar Papers

TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning

88% match1 repo32026

Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss

87% match2024

Pre-Trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness

87% match1 repo252026

Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models

86% match1 repo1112023

Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning

86% match1 repo142024

Difference Vector Equalization for Robust Fine-tuning of Vision-Language Models

86% match2025

Cite this paper

@article{yu2026select,
  title  = {Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models},
  author = {Yu-Chu Yu and Chi-Pin Huang and Jr-Jen Chen and Kai-Po Chang and Yung-Hsuan Lai and Fu-En Yang and Yu-Chiang Frank Wang},
  year   = {2026},
  eprint = {2403.09296},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2403.09296},
  journal = {ECCV 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.09296.svg)](https://codeofpaper.com/paper/2403.09296)

Discussion