RIBBON: Cost-Effective and QoS-Aware Deep Learning Model Inference using a Diverse Pool of Cloud Computing Instances

Baolin Li, Rohan Basu Roy, Tirthak Patel, Vijay Gadepally, Karen Gettings +1 more
7/23/2022
cs.DCcs.LG

No official code found. Best implementation: boringlee24/SC21_Ribbon (8 stars, MIT, last commit 2022-05-06).

Abstract

Deep learning model inference is a key service in many businesses and scientific discovery processes. This paper introduces RIBBON, a novel deep learning inference serving system that meets two competing objectives: quality-of-service (QoS) target and cost-effectiveness. The key idea behind RIBBON is to intelligently employ a diverse set of cloud computing instances (heterogeneous instances) to meet the QoS target and maximize cost savings. RIBBON devises a Bayesian Optimization-driven strategy that helps users build the optimal set of heterogeneous instances for their model inference service needs on cloud computing platforms -- and, RIBBON demonstrates its superiority over existing approaches of inference serving systems using homogeneous instance pools. RIBBON saves up to 16% of the inference service cost for different learning models including emerging deep learning recommender system models and drug-discovery enabling models.

View on arXivView PDF

Code Implementations(1)

boringlee24/SC21_RibbonCommunity73%
MIT

RIBBON: cost-effective and qos-aware deep learning model inference using a diverse pool of cloud computing instances

83Shell, PythonJun 1, 20214 years agoMIT

Similar Papers

IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency

84% match1 repo132023

Federated Learning While Providing Model as a Service: Joint Training and Inference Optimization

83% match2023

Lodestar: An Online-Learning LLM Inference Router

82% match2026

On the Cost of Model-Serving Frameworks: An Experimental Evaluation

82% match2024

Local-Cloud Inference Offloading for LLMs in Multi-Modal, Multi-Task, Multi-Dialogue Settings

82% match1 repo52025

XAIport: A Service Framework for the Early Adoption of XAI in AI Model Development

81% match2024

Cite this paper

@article{li2022ribbon,
  title  = {RIBBON: Cost-Effective and QoS-Aware Deep Learning Model Inference using a Diverse Pool of Cloud Computing Instances},
  author = {Baolin Li and Rohan Basu Roy and Tirthak Patel and Vijay Gadepally and Karen Gettings and Devesh Tiwari},
  year   = {2022},
  eprint = {2207.11434},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2207.11434v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2207.11434.svg)](https://codeofpaper.com/paper/2207.11434)

Discussion