Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

Yuheng Lu, Bingshuo Qian, Caixia Yuan, Huixing Jiang, Xiaojie Wang
10/22/2024
cs.CLcs.AI

No official code found. Best implementation: sutakori/CLoRA (2 stars, MIT, last commit 2025-07-24). 1 more below.

Abstract

Large language models (LLMs) exhibit remarkable capabilities in natural language processing but face catastrophic forgetting when learning new tasks, where adaptation to a new domain leads to a substantial decline in performance on previous tasks. In this paper, we propose Controlled LoRA (CLoRA), a sub-space regularization method on LoRA structure. Aiming to reduce the scale of output change while introduce minimal constraint on model capacity, CLoRA imposes constraint on the direction of updating matrix's null space. Experimental results on one-stage LLM finetuning tasks and continual learning settings highlight the superority of CLoRA as a effective parameter efficient finetuning method with catastrophic forgetting mitigating.Further investigation for model parameters indicates that CLoRA effectively balances the trade-off between model capacity and degree of forgetting.

View on arXivView PDF

Code Implementations(2)

OddFunction0205/clora-Community69%

《Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models》这篇论文的复现

01Shell, PythonJul 8, 20251 years ago
sutakori/CLoRACommunity67%
MIT

Code for ACL 2025 paper "Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models"

21PythonMay 27, 20251 years agoMIT

Similar Papers

CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation

91% match1 repo532024

SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA

90% match2025

CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models

89% match2024

Enhancing Parameter Efficiency and Generalization in Large-Scale Models: A Regularized and Masked Low-Rank Adaptation Approach

89% match2024

Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledge

89% match1 repo92025

LoRA Learns Less and Forgets Less

89% match3 repos222024

Cite this paper

@article{lu2024controlled,
  title  = {Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models},
  author = {Yuheng Lu and Bingshuo Qian and Caixia Yuan and Huixing Jiang and Xiaojie Wang},
  year   = {2024},
  eprint = {2410.16801},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.16801v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.16801.svg)](https://codeofpaper.com/paper/2410.16801)

Discussion