MuonAll: Muon Variant for Efficient Finetuning of Large Language Models

Saurabh Page, Advait Joshi, S. S. Sonawane
11/8/2025
cs.CLcs.LG

Official code: Saurabh750/optimizer (7 stars, MIT, last commit 2025-10-01).

Abstract

Muon optimizer has demonstrated robust results in pretraining of language models but its performance in finetuning of existing public pretrained models is not yet explored. Currently, Muon is used along with AdamW introducing a scope of improvement for adopting all parameters inside Muon. We introduce MuonAll, which incorporates all the parameters inside Muon by transforming into 2D matrices. We conduct extensive finetuning experiments across publicly available language models with model sizes upto half billion parameters. Muon and MuonAll perform at par with AdamW across major benchmarks, highlighting their effectiveness as alternative optimizers. We open-source the distributed implementations of Muon and MuonAll, available at https://github.com/Saurabh750/optimizer

View on arXivView PDF

Code Implementations(1)

Saurabh750/optimizerOfficial100%
MIT

Distributed Muon & MuonAll optimizer implementations which can be directly integrated with huggingface transformers library. The optimizers can be used for pretraining and finetuning language models.

70PythonAug 3, 20251 years agoMIT

Similar Papers

Muon is Scalable for LLM Training

87% match2 repos2.7k2025

NorMuon: Making Muon more efficient and scalable

85% match2025

Towards a Principled Muon under $μ\mathsf{P}$: Ensuring Spectral Conditions throughout Training

83% match2026

Convergence Bound and Critical Batch Size of Muon Optimizer

83% match2025

MARS-M: When Variance Reduction Meets Matrices

83% match1 repo7222025

REG: A Regularization Optimizer for Robust Training Dynamics

83% match2025

Cite this paper

@article{page2025muonall,
  title  = {MuonAll: Muon Variant for Efficient Finetuning of Large Language Models},
  author = {Saurabh Page and Advait Joshi and S. S. Sonawane},
  year   = {2025},
  eprint = {2511.06086},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2511.06086v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2511.06086.svg)](https://codeofpaper.com/paper/2511.06086)

Discussion