LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models

Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu
10/9/2023
cs.CLcs.LG

Official code: microsoft/LLMLingua (6,463 stars, MIT, last commit 2026-04-08).

Abstract

Large language models (LLMs) have been applied in various applications due to their astonishing capabilities. With advancements in technologies such as chain-of-thought (CoT) prompting and in-context learning (ICL), the prompts fed to LLMs are becoming increasingly lengthy, even exceeding tens of thousands of tokens. To accelerate model inference and reduce cost, this paper presents LLMLingua, a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity under high compression ratios, a token-level iterative compression algorithm to better model the interdependence between compressed contents, and an instruction tuning based method for distribution alignment between language models. We conduct experiments and analysis over four datasets from different scenarios, i.e., GSM8K, BBH, ShareGPT, and Arxiv-March23; showing that the proposed approach yields state-of-the-art performance and allows for up to 20x compression with little performance loss. Our code is available at https://aka.ms/LLMLingua.

View on arXivView PDF

Code Implementations(1)

microsoft/LLMLinguaOfficial100%
MIT

[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

6,463403Shell, PythonJul 7, 20236 months agoMIT

Similar Papers

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

90% match2 repos6.5k2023

Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt

89% match2023

LanguaShrink: Reducing Token Overhead with Psycholinguistics

88% match2024

Dynamic Compressing Prompts for Efficient Inference of Large Language Models

88% match2025

LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment

88% match1 repo282026

An Empirical Study on Prompt Compression for Large Language Models

88% match1 repo2922025

Cite this paper

@article{jiang2023llmlingua,
  title  = {LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models},
  author = {Huiqiang Jiang and Qianhui Wu and Chin-Yew Lin and Yuqing Yang and Lili Qiu},
  year   = {2023},
  eprint = {2310.05736},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.05736v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.05736.svg)](https://codeofpaper.com/paper/2310.05736)

Discussion