On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference

Siyu Ren, Kenny Q. Zhu
2/9/2024
cs.CLcs.AI

Official code: DRSY/EasyKV (62 stars, last commit 2024-02-13).

Abstract

Despite the recent success associated with Large Language Models (LLMs), they are notably cost-prohibitive to deploy in resource-constrained environments due to their excessive memory and computational demands. In addition to model parameters, the key-value cache is also stored in GPU memory, growing linearly with batch size and sequence length. As a remedy, recent works have proposed various eviction policies for maintaining the overhead of key-value cache under a given budget. This paper embarks on the efficacy of existing eviction policies in terms of importance score calculation and eviction scope construction. We identify the deficiency of prior policies in these two aspects and introduce RoCo, a robust cache omission policy based on temporal attention scores and robustness measures. Extensive experimentation spanning prefilling and auto-regressive decoding stages validates the superiority of RoCo. Finally, we release EasyKV, a versatile software package dedicated to user-friendly key-value constrained generative inference. Code available at https://github.com/DRSY/EasyKV.

View on arXivView PDF

Code Implementations(1)

DRSY/EasyKVOfficial100%

Easy control for Key-Value Constrained Generative LLM Inference(https://arxiv.org/abs/2402.06262)

625PythonJan 14, 20242 years ago
cache-evictioncache-managementkv-cachellm

Similar Papers

Learning to Evict from Key-Value Cache

88% match2026

Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference

86% match2 repos1.1k2024

Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity

86% match2025

NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time

86% match1 repo1.8k2024

ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution

86% match2026

LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation

86% match1 repo132026

Cite this paper

@article{ren2024efficacy,
  title  = {On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference},
  author = {Siyu Ren and Kenny Q. Zhu},
  year   = {2024},
  eprint = {2402.06262},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.06262v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.06262.svg)](https://codeofpaper.com/paper/2402.06262)

Discussion