Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction

Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha +5 more
4/12/2024
cs.DCcs.CLcs.LG

No official code found. Best implementation: James-QiuHaoran/LLM-serving-with-proxy-models (52 stars, Apache-2.0, last commit 2024-06-01).

Abstract

Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths. Our open-source SSJF implementation does not require changes to memory management or batching strategies. Evaluations on real-world datasets and production workload traces show that SSJF reduces average job completion times by 30.5-39.6% and increases throughput by 2.2-3.6x compared to FCFS schedulers, across no batching, dynamic batching, and continuous batching settings.

View on arXivView PDF

Code Implementations(1)

James-QiuHaoran/LLM-serving-with-proxy-modelsCommunity100%
Apache-2.0

Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction | A tiny BERT model can tell you the verbosity of an LLM (with low latency overhead!)

528Shell, PythonApr 12, 20242 years agoApache-2.0
bert-modelllamallm-inferencellm-servingproxy-model+1 more

Similar Papers

Efficient LLM Scheduling by Learning to Rank

90% match2 repos812024

Don't Stop Me Now: Embedding Based Scheduling for LLMs

87% match2024

Fast Inference for Augmented Large Language Models

86% match1 repo2024

PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference

85% match2024

Adaptively Robust LLM Inference Optimization under Prediction Uncertainty

84% match2025

Optimal Scheduling Algorithms for LLM Inference: Theory and Practice

83% match2025

Cite this paper

@article{qiu2024efficient,
  title  = {Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction},
  author = {Haoran Qiu and Weichao Mao and Archit Patke and Shengkun Cui and Saurabh Jha and Chen Wang and Hubertus Franke and Zbigniew T. Kalbarczyk and Tamer Başar and Ravishankar K. Iyer},
  year   = {2024},
  eprint = {2404.08509},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.08509v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.08509.svg)](https://codeofpaper.com/paper/2404.08509)

Discussion