QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron +4 more
3/30/2024
cs.LG

Official code: spcl/QuaRot (533 stars, Apache-2.0, last commit 2024-11-26).

Abstract

We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4 bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLaMa2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless 6 and 8 bit LLaMa2 models without any calibration data using round-to-nearest quantization. Code is available at: https://github.com/spcl/QuaRot.

View on arXivView PDF

Code Implementations(1)

spcl/QuaRotOfficial100%
Apache-2.0

Code for Neurips24 paper: QuaRot, an end-to-end 4-bit inference of large language models.

53378C, C++Mar 31, 20241 years agoApache-2.0

Similar Papers

SpinQuant: LLM quantization with learned rotations

86% match1 repo4382026

SpinQuant: LLM quantization with learned rotations

86% match1 repo4382024

RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations

84% match2025

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

83% match1 repo4332024

iFairy: the First 2-bit Complex LLM with All Parameters in $\{\pm1, \pm i\}$

82% match2025

Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMs

82% match2025

Cite this paper

@article{ashkboos2024quarot,
  title  = {QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs},
  author = {Saleh Ashkboos and Amirkeivan Mohtashami and Maximilian L. Croci and Bo Li and Pashmina Cameron and Martin Jaggi and Dan Alistarh and Torsten Hoefler and James Hensman},
  year   = {2024},
  eprint = {2404.00456},
  archivePrefix = {arXiv},
  doi    = {10.48550/arXiv.2404.00456},
  url    = {http://arxiv.org/abs/2404.00456v2},
  journal = {NEURIPS 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.00456.svg)](https://codeofpaper.com/paper/2404.00456)

Discussion