Efficient LLM Inference on CPUs

Haihao Shen, Hanwen Chang, Bo Dong, Yu Luo, Hengyu Meng
11/1/2023
cs.LGcs.AIcs.CL

Official code: intel/intel-extension-for-transformers (2,177 stars, Apache-2.0, last commit 2024-10-08).

Abstract

Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. However, deploying these models has been challenging due to the astronomical amount of model parameters, which requires a demand for large memory capacity and high memory bandwidth. In this paper, we propose an effective approach that can make the deployment of LLMs more efficiently. We support an automatic INT4 weight-only quantization flow and design a special LLM runtime with highly-optimized kernels to accelerate the LLM inference on CPUs. We demonstrate the general applicability of our approach on popular LLMs including Llama2, Llama, GPT-NeoX, and showcase the extreme inference efficiency on CPUs. The code is publicly available at: https://github.com/intel/intel-extension-for-transformers.

View on arXivView PDF

Code Implementations(1)

intel/intel-extension-for-transformersOfficial100%
Apache-2.0

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

2,177216C, C++Nov 11, 20222 years agoApache-2.0
4-bitsautoroundchatbotchatpdfgaudi3+11 more

Similar Papers

Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs

90% match1 repo131.9k2024

Inference Acceleration for Large Language Models on CPUs

88% match2024

LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

87% match1 repo632024

xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference

87% match3 repos2.2k2025

LLM Inference Unveiled: Survey and Roofline Model Insights

87% match1 repo6502024

A Comprehensive Study on Quantization Techniques for Large Language Models

87% match2024

Cite this paper

@article{shen2023efficient,
  title  = {Efficient LLM Inference on CPUs},
  author = {Haihao Shen and Hanwen Chang and Bo Dong and Yu Luo and Hengyu Meng},
  year   = {2023},
  eprint = {2311.00502},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.00502v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.00502.svg)](https://codeofpaper.com/paper/2311.00502)

Discussion