Orthogonal Subspace Learning for Language Model Continual Learning

Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao +4 more
10/22/2023
cs.CLcs.LG

Official code: cmnfriend/O-LoRA (217 stars, MIT, last commit 2024-07-13).

Abstract

Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.

View on arXivView PDF

Code Implementations(1)

cmnfriend/O-LoRAOfficial100%
MIT
21731Shell, PythonJun 11, 20232 years agoMIT

Similar Papers

Merge before Forget: A Single LoRA Continual Learning via Continual Merging

89% match2025

Is Parameter Collision Hindering Continual Learning in LLMs?

89% match1 repo382024

Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models

88% match2025

Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning

88% match1 repo132025

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

88% match2 repos22024

ELLA: Efficient Lifelong Learning for Adapters in Large Language Models

87% match2026

Cite this paper

@article{wang2023orthogonal,
  title  = {Orthogonal Subspace Learning for Language Model Continual Learning},
  author = {Xiao Wang and Tianze Chen and Qiming Ge and Han Xia and Rong Bao and Rui Zheng and Qi Zhang and Tao Gui and Xuanjing Huang},
  year   = {2023},
  eprint = {2310.14152},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.14152v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.14152.svg)](https://codeofpaper.com/paper/2310.14152)

Discussion