Iterative Orthogonalization Scaling Laws

Devan Selvaraj
5/6/2025
cs.LG

No confident code implementation found for this paper.

Abstract

The muon optimizer has picked up much attention as of late as a possible replacement to the seemingly omnipresent Adam optimizer. Recently, care has been taken to document the scaling laws of hyper-parameters under muon such as weight decay and learning rate. However, at much larger scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with scale. This paper shows this scaling behavior theoretically and empirically on random matrices but does not suggest what to do about it.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Cite this paper

@article{selvaraj2025iterative,
  title  = {Iterative Orthogonalization Scaling Laws},
  author = {Devan Selvaraj},
  year   = {2025},
  eprint = {2505.04005},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.04005v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.04005.svg)](https://codeofpaper.com/paper/2505.04005)

Discussion