Class Incremental Learning with Pre-trained Vision-Language Models

Xialei Liu, Xusheng Cao, Haori Lu, Jia-wen Xiao, Andrew D. Bagdanov +1 more
10/31/2023
cs.CVcs.LG

Official code: sun-hailong/CVPR24-Ease (83 stars, PyTorch, last commit 2025-04-01). 3 more implementations below.

Abstract

With the advent of large-scale pre-trained models, interest in adapting and exploiting them for continual learning scenarios has grown. In this paper, we propose an approach to exploiting pre-trained vision-language models (e.g. CLIP) that enables further adaptation instead of only using zero-shot learning of new tasks. We augment a pre-trained CLIP model with additional layers after the Image Encoder or before the Text Encoder. We investigate three different strategies: a Linear Adapter, a Self-attention Adapter, each operating on the image embedding, and Prompt Tuning which instead modifies prompts input to the CLIP text encoder. We also propose a method for parameter retention in the adapter layers that uses a measure of parameter importance to better maintain stability and plasticity during incremental learning. Our experiments demonstrate that the simplest solution -- a single Linear Adapter layer with parameter retention -- produces the best results. Experiments on several conventional benchmarks consistently show a significant margin of improvement over the current state-of-the-art.

View on arXivView PDF

Code Implementations(4)

sun-hailong/CVPR24-EaseOfficial100%
PyTorch

[CVPR 2024] The code repository for "Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning" in PyTorch.

834PythonMar 15, 20241 years ago
continual-learningdeep-learningincremental-learninglifelong-learningmachine-learning+3 more
sun-hailong/AAAI25-MOSOfficial100%

[AAAI 2025] The code repository for "MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning" in PyTorch.

579PythonDec 10, 20241 years ago
OatmealLiu/MSc-iNCDCommunity100%
MIT

[ICPR'24 Oral] Large-scale Pre-trained Models are Surprisingly Strong in Incremental Novel Class Discovery

432Shell, PythonMar 28, 20232 years agoMIT
LAMDA-CL/RevisitingCILCommunity75%

Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need (IJCV 2024)

15321PythonMar 13, 20232 years ago
catastrophic-forgettingclass-incremental-learningcontinual-learningincremental-learninglifelong-learning

Similar Papers

Multi-Modal Adapter for Vision-Language Models

88% match2 repos1012024

Learning without Forgetting for Vision-Language Models

87% match1 repo642023

Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusion

87% match1 repo512024

Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models

87% match1 repo1112023

Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters

87% match1 repo2752026

Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model

86% match1 repo72024

Cite this paper

@article{liu2023class,
  title  = {Class Incremental Learning with Pre-trained Vision-Language Models},
  author = {Xialei Liu and Xusheng Cao and Haori Lu and Jia-wen Xiao and Andrew D. Bagdanov and Ming-Ming Cheng},
  year   = {2023},
  eprint = {2310.20348},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.20348v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.20348.svg)](https://codeofpaper.com/paper/2310.20348)

Discussion