Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation

Yongfu Xue
10/4/2025
cs.LGcs.CL

No official code found. Best implementation: xueyongfu11/IniLoRA (1 star, last commit 2026-01-09).

Abstract

The rapid development of parameter-efficient fine-tuning methods has noticeably improved the efficiency of adapting large language models. Among these, LoRA has gained widespread popularity due to its strong balance of effectiveness and parameter efficiency. However, LoRA relies on initializing two low-rank matrices whose product is zero, which limits its ability to effectively activate and leverage the original model weights-creating a potential bottleneck for optimal performance. To address this limitation, we propose \textbf{IniLoRA}, a novel initialization strategy that initializes the low-rank matrices to closely approximate the original model weights. Experimental results indicate that IniLoRA achieves better performance than LoRA across a range of models and tasks. Additionally, we introduce two variants, IniLoRA-$α$ and IniLoRA-$β$, both leveraging distinct initialization methods to enhance performance further.

View on arXivView PDF

Code Implementations(1)

xueyongfu11/IniLoRACommunity65%

Paper: Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation

10Shell, PythonJan 9, 20269 months ago

Similar Papers

Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation

100% match2025

Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning

91% match1 repo512024

LoRA-Mini : Adaptation Matrices Decomposition and Selective Training

89% match1 repo22024

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning

88% match2024

ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition

88% match2026

Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models

88% match2025

Cite this paper

@article{xue2025optimizing,
  title  = {Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation},
  author = {Yongfu Xue},
  year   = {2025},
  eprint = {2510.03731},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.03731v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.03731.svg)](https://codeofpaper.com/paper/2510.03731)

Discussion