LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

Krishna Teja Chitty-Venkata, Siddhisanket Raskar, Bharat Kale, Farah Ferdaus, Aditya Tanikanti +4 more
10/31/2024
cs.LG

No official code found. Best implementation: argonne-lcf/LLM-Inference-Bench (63 stars, BSD-3-Clause, last commit 2025-07-18).

Abstract

Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. However, the computational demands of these complex models pose significant challenges, requiring efficient hardware acceleration. Benchmarking the performance of LLMs across diverse hardware platforms is crucial to understanding their scalability and throughput characteristics. We introduce LLM-Inference-Bench, a comprehensive benchmarking suite to evaluate the hardware inference performance of LLMs. We thoroughly analyze diverse hardware platforms, including GPUs from Nvidia and AMD and specialized AI accelerators, Intel Habana and SambaNova. Our evaluation includes several LLM inference frameworks and models from LLaMA, Mistral, and Qwen families with 7B and 70B parameters. Our benchmarking results reveal the strengths and limitations of various models, hardware platforms, and inference frameworks. We provide an interactive dashboard to help identify configurations for optimal performance for a given hardware platform.

View on arXivView PDF

Code Implementations(1)

argonne-lcf/LLM-Inference-BenchCommunity100%
BSD-3-Clause

LLM-Inference-Bench

637Shell, PythonJul 29, 20241 years agoBSD-3-Clause
benchmarkdeepspeedinferencellamacppllm+2 more

Similar Papers

Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective

92% match1 repo182024

LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services

91% match1 repo232024

Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models

90% match1 repo1252024

AI Accelerators for Large Language Model Inference: Architecture Analysis and Scaling Strategies

90% match2025

Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling

90% match2025

Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models

88% match2023

Cite this paper

@article{chittyvenkata2024llminferencebench,
  title  = {LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators},
  author = {Krishna Teja Chitty-Venkata and Siddhisanket Raskar and Bharat Kale and Farah Ferdaus and Aditya Tanikanti and Ken Raffenetti and Valerie Taylor and Murali Emani and Venkatram Vishwanath},
  year   = {2024},
  eprint = {2411.00136},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.00136v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.00136.svg)](https://codeofpaper.com/paper/2411.00136)

Discussion