Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models

Kefan Cao, Shuaicheng Wu
9/7/2025
cs.CL

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

Large language models (LLMs) suffer from catastrophic forgetting in sequential multi-task learning. Existing parameter regularization methods (e.g., O-LoRA, N-LoRA) mitigate interference via low-rank subspace orthogonality, but additive updates distort the intrinsic geometry of model parameters. We propose \textbf{OLieRA}, a Lie group based fine-tuning framework that preserves parameter geometry through multiplicative updates while enforcing orthogonality across task subspaces. OLieRA achieves state-of-the-art performance on the Standard CL benchmark and remains highly competitive under large task sequences. It further inherits the replay-free and task-ID free inference properties of O-LoRA, establishing a principled paradigm for continual learning in LLMs.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Orthogonal Subspace Learning for Language Model Continual Learning

88% match1 repo2172023

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

87% match2 repos22024

Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning

87% match1 repo132025

Is Parameter Collision Hindering Continual Learning in LLMs?

87% match1 repo382024

CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation

86% match1 repo532024

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

86% match2024

Cite this paper

@article{cao2025orthogonal,
  title  = {Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models},
  author = {Kefan Cao and Shuaicheng Wu},
  year   = {2025},
  eprint = {2509.06100},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2509.06100v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2509.06100.svg)](https://codeofpaper.com/paper/2509.06100)

Discussion