Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

Chenxu Wang, Yilin Lyu, Zicheng Sun, Liping Jing
7/3/2025
cs.LGcs.AIcs.CE

Official code: Wcxwcxw/GORP (7 stars, last commit 2025-10-29).

Abstract

Continual fine-tuning of Large Language Models (LLMs) is hampered by the trade-off between efficiency and expressiveness. Low-Rank Adaptation (LoRA) offers efficiency but constrains the model's ability to learn new tasks and transfer knowledge due to its low-rank nature and reliance on explicit parameter constraints. We propose GORP (Gradient LOw Rank Projection) for Continual Learning, a novel training strategy that overcomes these limitations by synergistically combining full and low-rank parameters and jointly updating within a unified low-rank gradient subspace. GORP expands the optimization space while preserving efficiency and mitigating catastrophic forgetting. Extensive experiments on continual learning benchmarks demonstrate GORP's superior performance compared to existing state-of-the-art approaches. Code is available at https://github.com/Wcxwcxw/GORP.

View on arXivView PDF

Code Implementations(1)

Wcxwcxw/GORPOfficial100%

Official Code for Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

73Shell, PythonMay 26, 202511 months ago

Similar Papers

AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning

88% match1 repo2024

AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections

87% match2025

Gradient Weight-normalized Low-rank Projection for Efficient LLM Training

87% match1 repo62024

Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs

87% match2024

Sparse Low-rank Adaptation of Pre-trained Language Models

86% match1 repo872023

ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning

86% match2025

Cite this paper

@article{wang2025continual,
  title  = {Continual Gradient Low-Rank Projection Fine-Tuning for LLMs},
  author = {Chenxu Wang and Yilin Lyu and Zicheng Sun and Liping Jing},
  year   = {2025},
  eprint = {2507.02503},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2507.02503v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2507.02503.svg)](https://codeofpaper.com/paper/2507.02503)

Discussion