FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design

Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao +8 more
1/25/2024
cs.LGcs.AIcs.AR

Official code: usyd-fsalab/fp6_llm (281 stars, Apache-2.0, last commit 2025-07-16). 1 more implementation below.

Abstract

Six-bit quantization (FP6) can effectively reduce the size of large language models (LLMs) and preserve the model quality consistently across varied applications. However, existing systems do not provide Tensor Core support for FP6 quantization and struggle to achieve practical performance improvements during LLM inference. It is challenging to support FP6 quantization on GPUs due to (1) unfriendly memory access of model weights with irregular bit-width and (2) high runtime overhead of weight de-quantization. To address these problems, we propose TC-FPx, the first full-stack GPU kernel design scheme with unified Tensor Core support of float-point weights for various quantization bit-width. We integrate TC-FPx kernel into an existing inference system, providing new end-to-end support (called FP6-LLM) for quantized LLM inference, where better trade-offs between inference cost and model quality are achieved. Experiments show that FP6-LLM enables the inference of LLaMA-70b using only a single GPU, achieving 1.69x-2.65x higher normalized inference throughput than the FP16 baseline. The source code is publicly available at https://github.com/usyd-fsalab/fp6_llm.

View on arXivView PDF

Code Implementations(2)

usyd-fsalab/fp6_llmOfficial100%
Apache-2.0

An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).

28126C, C++Mar 4, 20241 years agoApache-2.0
deepspeedai/DeepSpeedCommunity100%
PyTorchApache-2.0

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

43,2295,039C, C++Jan 23, 20205 months agoApache-2.0
billion-parameterscompressiondata-parallelismdeep-learninggpu+8 more

Similar Papers

FlexQ: Efficient Post-training INT6 Quantization for LLM Serving via Algorithm-System Co-Design

90% match1 repo112025

FPGA Co-Design for Efficient N:M Sparse and Quantized Model Inference

85% match2025

Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models

85% match2023

Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores

85% match2024

FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization

85% match2024

ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats

85% match1 repo43.2k2023

Cite this paper

@article{xia2024fpllm,
  title  = {FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design},
  author = {Haojun Xia and Zhen Zheng and Xiaoxia Wu and Shiyang Chen and Zhewei Yao and Stephen Youn and Arash Bakhtiari and Michael Wyatt and Donglin Zhuang and Zhongzhu Zhou and Olatunji Ruwase and Yuxiong He and Shuaiwen Leon Song},
  year   = {2024},
  eprint = {2401.14112},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.14112v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.14112.svg)](https://codeofpaper.com/paper/2401.14112)

Discussion