VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

Yixuan Wang, Qingyu Shi, Jiayu Zhou, Dianbo Liu, Ziwei He +1 more
3/17/2026
cs.CL

No official code found. Best implementation: LUMIA-Group/VQKV (7 stars, MIT, last commit 2026-03-18).

Abstract

The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8\% compression ratio on LLaMA3.1-8B while retaining 98.6\% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.

View on arXivView PDF

Code Implementations(1)

LUMIA-Group/VQKVCommunity25%
MIT

The official repo of paper: VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

72PythonMar 15, 20267 months agoMIT

Similar Papers

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

87% match1 repo212026

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

86% match2026

Q-MambaIR: Accurate Quantized Mamba for Efficient Image Restoration

85% match2025

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

85% match1 repo212025

When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models

85% match2025

Continuous Approximations for Improving Quantization Aware Training of LLMs

84% match2024

Cite this paper

@article{wang2026vqkv,
  title  = {VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization},
  author = {Yixuan Wang and Qingyu Shi and Jiayu Zhou and Dianbo Liu and Ziwei He and Zhouhan Lin},
  year   = {2026},
  eprint = {2603.16435},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2603.16435v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2603.16435.svg)](https://codeofpaper.com/paper/2603.16435)

Discussion