CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation

Muhammad Fawi
8/26/2024
cs.LGcs.AIcs.CL

No official code found. Best implementation: MNoorFawi/curlora (53 stars, MIT, last commit 2024-08-28).

Abstract

This paper introduces CURLoRA, a novel approach to fine-tuning large language models (LLMs) that leverages CUR matrix decomposition in the context of Low-Rank Adaptation (LoRA). Our method addresses two critical challenges in LLM fine-tuning: mitigating catastrophic forgetting during continual learning and reducing the number of trainable parameters. We propose a unique modification to the CUR decomposition process, utilizing inverted probabilities for column and row selection which acts as an implicit regularization, and initializing the $U$ matrix as a zero matrix, and only fine-tuning it. We demonstrate through experiments on multiple datasets that CURLoRA outperforms standard LoRA in mitigating catastrophic forgetting. It maintains model stability and performance across tasks while significantly reducing the number of trainable parameters. Our results show that CURLoRA achieves very good and stable task accuracy while maintaining base model's perplexity scores fixed compared to LoRA upon continual fine-tuning, particularly in scenarios with limited data.

View on arXivView PDF

Code Implementations(1)

MNoorFawi/curloraCommunity100%
MIT

The code repository for the CURLoRA research paper. Stable LLM continual fine-tuning and catastrophic forgetting mitigation.

536Python, Jupyter NotebookJul 10, 20242 years agoMIT
aicatastrophic-forgettingcontinual-learningfine-tuninggenerative-ai+2 more

Similar Papers

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

91% match2 repos22024

Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning

88% match1 repo392024

STABLE: Gated Continual Learning for Large Language Models

87% match1 repo22025

LoRA-Mini : Adaptation Matrices Decomposition and Selective Training

87% match1 repo22024

SECURA: Sigmoid-Enhanced CUR Decomposition with Uninterrupted Retention and Low-Rank Adaptation in Large Language Models

87% match2025

PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization

87% match2024

Cite this paper

@article{fawi2024curlora,
  title  = {CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation},
  author = {Muhammad Fawi},
  year   = {2024},
  eprint = {2408.14572},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.14572v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.14572.svg)](https://codeofpaper.com/paper/2408.14572)

Discussion