On the Spatial Structure of Mixture-of-Experts in Transformers

Daniel Bershatsky, Ivan Oseledets
4/6/2025
cs.CLcs.AIcs.LG

No confident code implementation found for this paper.

Abstract

A common assumption is that MoE routers primarily leverage semantic features for expert selection. However, our study challenges this notion by demonstrating that positional token information also plays a crucial role in routing decisions. Through extensive empirical analysis, we provide evidence supporting this hypothesis, develop a phenomenological explanation of the observed behavior, and discuss practical implications for MoE-based architectures.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models

86% match2024

Towards an empirical understanding of MoE design choices

86% match2024

Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Models

83% match2025

Probing Semantic Routing in Large Mixture-of-Expert Models

83% match2025

Routers in Vision Mixture of Experts: An Empirical Study

82% match2024

Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition

82% match2025

Cite this paper

@article{bershatsky2025spatial,
  title  = {On the Spatial Structure of Mixture-of-Experts in Transformers},
  author = {Daniel Bershatsky and Ivan Oseledets},
  year   = {2025},
  eprint = {2504.04444},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2504.04444v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2504.04444.svg)](https://codeofpaper.com/paper/2504.04444)

Discussion