Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models

Shuming Shi, Enbo Zhao, Deng Cai, Leyang Cui, Xinting Huang +1 more
1/16/2024
cs.CL

No official code found. Best implementation: inferflow/inferflow (250 stars, MIT, last commit 2024-03-15).

Abstract

We present Inferflow, an efficient and highly configurable inference engine for large language models (LLMs). With Inferflow, users can serve most of the common transformer models by simply modifying some lines in corresponding configuration files, without writing a single line of source code. Compared with most existing inference engines, Inferflow has some key features. First, by implementing a modular framework of atomic build-blocks and technologies, Inferflow is compositionally generalizable to new models. Second, 3.5-bit quantization is introduced in Inferflow as a tradeoff between 3-bit and 4-bit quantization. Third, hybrid model partitioning for multi-GPU inference is introduced in Inferflow to better balance inference speed and throughput than the existing partition-by-layer and partition-by-tensor strategies.

View on arXivView PDF

Code Implementations(1)

inferflow/inferflowCommunity100%
MIT

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

25025C, C++Dec 26, 20232 years agoMIT
baichuan2bloomdeepseekfalcongemma+14 more

Similar Papers

InferCept: Efficient Intercept Support for Augmented Large Language Model Inference

87% match1 repo332024

Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference

84% match2024

A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency

84% match1 repo2372025

Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models

83% match2025

Towards Coarse-to-Fine Evaluation of Inference Efficiency for Large Language Models

83% match2024

Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations

83% match2024

Cite this paper

@article{shi2024inferflow,
  title  = {Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models},
  author = {Shuming Shi and Enbo Zhao and Deng Cai and Leyang Cui and Xinting Huang and Huayang Li},
  year   = {2024},
  eprint = {2401.08294},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.08294v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.08294.svg)](https://codeofpaper.com/paper/2401.08294)

Discussion