OmniKVQuant: KV Cache Quantization for Omni-LLMs

Suho Yoo, Hyunjong Ok, Jongmin Choi, Jihoo Jung, Joon Son Chung
9/10/2026
cs.CV

Official code: kaistmm/OmniKVQuant (2 stars, Apache-2.0, last commit 2026-09-10).

Abstract

As Omni-modal large language models (Omni-LLMs) take in audio, video and text together, their KV cache memory cost grows. KV cache quantization is the de facto approach in text-only LLMs, but its application to Omni-LLMs remains unexplored. In this paper, we analyze how TurboQuant, a representative rotation-based KV cache quantization method, behaves on multimodal caches and identify two critical issues: temporal key drift and heterogeneous value geometry. To address these, we propose OmniKVQuant, a training-free framework that (i) sets the key quantization range over each short window of the input stream; and (ii) rotates values separately per modality. On Qwen2.5-Omni and Qwen3-Omni, OmniKVQuant enables 2-bit KV caches while substantially preserving performance across seven audio-visual benchmarks. We further provide a fused Triton decode kernel that unpacks the 2-bit cache during attention, so no dense FP16 cache is ever built. Code: https://github.com/kaistmm/OmniKVQuant

View on arXivView PDF

Code Implementations(1)

kaistmm/OmniKVQuantOfficial100%
Apache-2.0

OmniKVQuant: KV Cache Quantization for Omni-LLMs

20PythonSep 10, 20261 months agoApache-2.0

Similar Papers

Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models

88% match2025

AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference

86% match2025

Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models

86% match1 repo312025

AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization

85% match2025

OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

84% match1 repo1422026

Efficient Inference of Vision Instruction-Following Models with Elastic Cache

83% match1 repo432026

Cite this paper

@article{yoo2026omnikvquant,
  title  = {OmniKVQuant: KV Cache Quantization for Omni-LLMs},
  author = {Suho Yoo and Hyunjong Ok and Jongmin Choi and Jihoo Jung and Joon Son Chung},
  year   = {2026},
  eprint = {2609.11582},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2609.11582v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2609.11582.svg)](https://codeofpaper.com/paper/2609.11582)

Discussion