Revisiting Multimodal Positional Encoding in Vision-Language Models

Jie Huang, Xuejing Liu, Sibo Song, Ruibing Hou, Hong Chang +2 more
10/27/2025
cs.CV

Official code: JJJYmmm/Multimodal-RoPEs (88 stars, MIT, last commit 2026-03-16).

Abstract

Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) by examining its two core components: position design and frequency allocation. Through extensive experiments, we identify three key guidelines: positional coherence, full frequency utilization, and preservation of textual priors-ensuring unambiguous layout, rich representation, and faithful transfer from the pre-trained LLM. Based on these insights, we propose Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and plug-and-play variants that require no architectural changes. Our methods consistently outperform existing approaches across diverse benchmarks, with significant improvements in both general and fine-grained multimodal understanding. Code will be avaliable at https://github.com/JJJYmmm/Multimodal-RoPEs.

View on arXivView PDF

Code Implementations(1)

JJJYmmm/Multimodal-RoPEsOfficial100%
MIT

Official implement of paper "Revisiting Multimodal Positional Encoding in Vision–Language Models", ICLR 2026

883PythonSep 26, 20257 months agoMIT

Similar Papers

OMEGA: Optimized Multimodal Position Encoding Index Derivation with Global Adaptive Scaling for Vision-Language Models

88% match2025

Xmodel-VLM: A Simple Baseline for Multimodal Vision Language Model

84% match1 repo672024

Law of Vision Representation in MLLMs

83% match1 repo1802024

V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding

83% match1 repo602024

HiMix: Reducing Computational Complexity in Large Vision-Language Models

82% match2025

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

82% match1 repo2.0k2026

Cite this paper

@article{huang2025revisiting,
  title  = {Revisiting Multimodal Positional Encoding in Vision-Language Models},
  author = {Jie Huang and Xuejing Liu and Sibo Song and Ruibing Hou and Hong Chang and Junyang Lin and Shuai Bai},
  year   = {2025},
  eprint = {2510.23095},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.23095v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.23095.svg)](https://codeofpaper.com/paper/2510.23095)

Discussion