MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang +6 more
5/20/2024
cs.CLcs.LG

No official code found. Best implementation: kongds/MoRA (362 stars, Apache-2.0, last commit 2024-08-07). 1 more below.

Abstract

Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge. Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters. To achieve it, we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix. Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA. We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.

View on arXivView PDF

Code Implementations(2)

kongds/MoRACommunity100%
Apache-2.0

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

36223PythonMay 13, 20242 years agoApache-2.0
sanowl/MoRA-High-Rank-Matrix-Adaptation-for-Parameter-Efficient-Fine-Tuning-of-Large-Language-ModelsCommunity71%
MIT
30Cython, PythonMay 27, 20241 years agoMIT

Similar Papers

LoRA-Mini : Adaptation Matrices Decomposition and Selective Training

90% match1 repo22024

ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning

90% match2025

Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning

90% match1 repo512024

Sparse Low-rank Adaptation of Pre-trained Language Models

90% match1 repo872023

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

90% match2024

LoRMA: Low-Rank Multiplicative Adaptation for LLMs

90% match2025

Cite this paper

@article{jiang2024mora,
  title  = {MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning},
  author = {Ting Jiang and Shaohan Huang and Shengyue Luo and Zihan Zhang and Haizhen Huang and Furu Wei and Weiwei Deng and Feng Sun and Qi Zhang and Deqing Wang and Fuzhen Zhuang},
  year   = {2024},
  eprint = {2405.12130},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.12130v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.12130.svg)](https://codeofpaper.com/paper/2405.12130)

Discussion