Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression

Michael R. Metel, Boxing Chen, Mehdi Rezagholizadeh
12/7/2024
cs.CL

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Several works have developed eviction policies to remove key-value (KV) pairs from the KV cache for more efficient inference. The focus has been on compressing the KV cache after the input prompt has been processed for faster token generation. In settings with limited GPU memory, and when the input context is longer than the generation length, we show that by also compressing the KV cache during the input processing phase, larger batch sizes can be used resulting in significantly higher throughput while still maintaining the original model's accuracy.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving

87% match2025

CORM: Cache Optimization with Recent Message for Large Language Model Inference

86% match2024

BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching

85% match2024

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

85% match1 repo372024

KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head

85% match1 repo1582024

RefreshKV: Updating Small KV Cache During Long-form Generation

85% match1 repo12024

Cite this paper

@article{metel2024batchmax,
  title  = {Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression},
  author = {Michael R. Metel and Boxing Chen and Mehdi Rezagholizadeh},
  year   = {2024},
  eprint = {2412.05693},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.05693v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.05693.svg)](https://codeofpaper.com/paper/2412.05693)

Discussion