Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

Mehran Salmani, Saeid Ghafouri, Alireza Sanaee, Kamran Razavi, Max Mühlhäuser +3 more
4/21/2023
cs.LGcs.DCeess.SY

No official code found. Best implementation: reconfigurable-ml-pipeline/InfAdapter (9 stars, last commit 2024-04-30).

Abstract

The use of machine learning (ML) inference for various applications is growing drastically. ML inference services engage with users directly, requiring fast and accurate responses. Moreover, these services face dynamic workloads of requests, imposing changes in their computing resources. Failing to right-size computing resources results in either latency service level objectives (SLOs) violations or wasted computing resources. Adapting to dynamic workloads considering all the pillars of accuracy, latency, and resource cost is challenging. In response to these challenges, we propose InfAdapter, which proactively selects a set of ML model variants with their resource allocations to meet latency SLO while maximizing an objective function composed of accuracy and cost. InfAdapter decreases SLO violation and costs up to 65% and 33%, respectively, compared to a popular industry autoscaler (Kubernetes Vertical Pod Autoscaler).

View on arXivView PDF

Code Implementations(1)

reconfigurable-ml-pipeline/InfAdapterCommunity100%

Source code of "Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems"

91Shell, PythonJan 11, 20232 years ago

Similar Papers

Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelines

81% match1 repo52024

Efficient Multi-stage Inference on Tabular Data

81% match2023

Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameters

80% match2026

Latenrgy: Model Agnostic Latency and Energy Consumption Prediction for Binary Classifiers

80% match2024

AutoScout: Structured Optimization for Automating ML System Configuration

80% match2026

Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference

80% match2024

Cite this paper

@article{salmani2023reconciling,
  title  = {Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems},
  author = {Mehran Salmani and Saeid Ghafouri and Alireza Sanaee and Kamran Razavi and Max Mühlhäuser and Joseph Doyle and Pooyan Jamshidi and Mohsen Sharifi},
  year   = {2023},
  eprint = {2304.10892},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.10892v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.10892.svg)](https://codeofpaper.com/paper/2304.10892)

Discussion