Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules

Yilun Liu, Yunpu Ma, Yuetian Lu, Shuo Chen, Zifeng Ding +1 more
8/4/2025
cs.LGcs.AIcs.CL

No official code found. Best implementation: liuyilun2000/PERFT (2 stars, last commit 2026-01-15).

Abstract

Mixture-of-Experts (MoE) benefits from a dynamic routing mechanism among their specialized experts, which existing Parameter- Efficient Fine-Tuning (PEFT) strategies fail to leverage. This motivates us to investigate whether adaptation modules themselves should incorporate routing mechanisms to align with MoE's multi-expert architecture. We analyze dynamics of core components when applying PEFT to MoE language models and examine how different routing strategies affect adaptation effectiveness. Extensive experiments adapting OLMoE-1B-7B and Mixtral-8x7B on various commonsense and math reasoning tasks validate the performance and efficiency of our routed approach. We identify the optimal configurations for different scenarios and provide empirical analyses with practical insights to facilitate better PEFT and MoE applications.

View on arXivView PDF

Code Implementations(1)

liuyilun2000/PERFTCommunity75%

Official repository for paper Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules

20Shell, PythonOct 2, 20249 months ago
llmmoepeft

Similar Papers

PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model

89% match2024

PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning

88% match1 repo62025

Rewiring Experts on the Fly:Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert models

86% match1 repo62025

Harder Tasks Need More Experts: Dynamic Routing in MoE Models

86% match1 repo742024

TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts

86% match1 repo52025

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models

85% match1 repo7512024

Cite this paper

@article{liu2025parameterefficient,
  title  = {Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules},
  author = {Yilun Liu and Yunpu Ma and Yuetian Lu and Shuo Chen and Zifeng Ding and Volker Tresp},
  year   = {2025},
  eprint = {2508.02587},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2508.02587v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2508.02587.svg)](https://codeofpaper.com/paper/2508.02587)

Discussion