Efficient LLM Scheduling by Learning to Rank

Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica +1 more
8/28/2024
cs.LG

Official code: hao-ai-lab/vllm-ltr (81 stars, Apache-2.0, last commit 2024-11-04). 1 more implementation below.

Abstract

In Large Language Model (LLM) inference, the output length of an LLM request is typically regarded as not known a priori. Consequently, most LLM serving systems employ a simple First-come-first-serve (FCFS) scheduling strategy, leading to Head-Of-Line (HOL) blocking and reduced throughput and service quality. In this paper, we reexamine this assumption -- we show that, although predicting the exact generation length of each request is infeasible, it is possible to predict the relative ranks of output lengths in a batch of requests, using learning to rank. The ranking information offers valuable guidance for scheduling requests. Building on this insight, we develop a novel scheduler for LLM inference and serving that can approximate the shortest-job-first (SJF) schedule better than existing approaches. We integrate this scheduler with the state-of-the-art LLM serving system and show significant performance improvement in several important applications: 2.8x lower latency in chatbot serving and 6.5x higher throughput in synthetic data generation. Our code is available at https://github.com/hao-ai-lab/vllm-ltr.git

View on arXivView PDF

Code Implementations(2)

hao-ai-lab/vllm-ltrOfficial100%
Apache-2.0

[NeurIPS 2024] Efficient LLM Scheduling by Learning to Rank

8117C, C++Oct 14, 20241 years agoApache-2.0
sanowl/Efficient-LLM-Scheduling-by-Learning-to-RankCommunity70%

This project implements an efficient scheduling system for Large Language Model (LLM) inference, as described in the paper "Efficient LLM Scheduling by Learning to Rank"

90PythonSep 3, 20241 years ago

Similar Papers

Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction

90% match1 repo522024

Don't Stop Me Now: Embedding Based Scheduling for LLMs

88% match2024

LLM Serving Optimization with Variable Prefill and Decode Lengths

87% match2025

Fast Inference for Augmented Large Language Models

86% match1 repo2024

PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference

85% match2024

Adaptively Robust LLM Inference Optimization under Prediction Uncertainty

85% match2025

Cite this paper

@article{fu2024efficient,
  title  = {Efficient LLM Scheduling by Learning to Rank},
  author = {Yichao Fu and Siqi Zhu and Runlong Su and Aurick Qiao and Ion Stoica and Hao Zhang},
  year   = {2024},
  eprint = {2408.15792},
  archivePrefix = {arXiv},
  doi    = {10.48550/arXiv.2408.15792},
  url    = {http://arxiv.org/abs/2408.15792v1},
  journal = {NEURIPS 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.15792.svg)](https://codeofpaper.com/paper/2408.15792)

Discussion