G-KV: Decoding-Time KV Cache Eviction with Global Attention

Mengqi Liao, Lu Wang, Chaoyun Zhang, Zekai Shen, Xiaowei Mao +5 more
11/29/2025
cs.CLcs.AI

Official code: microsoft/G-KV (4 stars, last commit 2026-04-08).

Abstract

Recent reasoning large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths. KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning. However, existing methods often focus on prompt compression or token eviction with local attention score, overlooking the long-term importance of tokens. We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance. Additionally, we introduce post-training techniques, including reinforcement learning and distillation, to optimize models for compressed KV cache settings. The code of this paper is available on: https://github.com/microsoft/G-KV.

View on arXivView PDF

Code Implementations(1)

microsoft/G-KVOfficial100%

G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance.

40Shell, PythonOct 9, 20256 months ago

Similar Papers

ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution

91% match2026

KVCompose: Efficient Structured KV Cache Compression with Composite Tokens

89% match2025

Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction

89% match1 repo12025

EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importance

88% match2024

Learning to Evict from Key-Value Cache

88% match2026

Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution

88% match1 repo2025

Cite this paper

@article{liao2025gkv,
  title  = {G-KV: Decoding-Time KV Cache Eviction with Global Attention},
  author = {Mengqi Liao and Lu Wang and Chaoyun Zhang and Zekai Shen and Xiaowei Mao and Si Qin and Qingwei Lin and Saravan Rajmohan and Dongmei Zhang and Huaiyu Wan},
  year   = {2025},
  eprint = {2512.00504},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2512.00504v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2512.00504.svg)](https://codeofpaper.com/paper/2512.00504)

Discussion