Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity

Yuji Byun, Jaeho Lee
6/25/2024
cs.DCcs.LG

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

Low-rank adaptation (LoRA) offers an efficient alternative to full-weight adaptation in federated fine-tuning of language models, significantly reducing computational costs. By adjusting ranks for each client, federated LoRA enables flexible resource allocation. However, we observe that heterogeneous ranks among clients lead to unstable performance. Our analysis attributes this instability to the conventional zero-padding aggregation strategy, which dilutes information from high-rank clients during model aggregation. To address this issue, we propose a replication-based padding strategy that better retains valuable information from clients with high-quality data. Empirically, this approach accelerates convergence and enhances the global model's predictive performance.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneity

90% match2026

AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption

90% match2025

FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Models

90% match1 repo2025

Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs

89% match1 repo282025

Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRA

89% match2025

Aggregating Low Rank Adapters in Federated Fine-tuning

89% match2025

Cite this paper

@article{byun2024towards,
  title  = {Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity},
  author = {Yuji Byun and Jaeho Lee},
  year   = {2024},
  eprint = {2406.17477},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.17477v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.17477.svg)](https://codeofpaper.com/paper/2406.17477)

Discussion