LQER: Low-Rank Quantization Error Reconstruction for LLMs

Cheng Zhang, Jianyi Cheng, George A. Constantinides, Yiren Zhao
2/4/2024
cs.LGcs.CL

Official code: ChengZhang-98/LQER (19 stars, last commit 2024-07-11).

Abstract

Post-training quantization of Large Language Models (LLMs) is challenging. In this work, we introduce Low-rank Quantization Error Reduction (LQER), which combines quantization and low-rank approximation to recover the model capability. LQER leverages an activation-induced scale matrix to drive the singular value distribution of quantization error towards a desirable distribution, which enables nearly-lossless W4A8 quantization on various LLMs and downstream tasks without the need for knowledge distillation, grid search, or gradient-base iterative optimization. Unlike existing methods, the computation pattern of LQER eliminates the need for specialized Scatter and Gather processes to collect high-precision weights from irregular memory locations. Our W4A8 LLMs achieve near-lossless performance on six popular downstream tasks, while using 1.36$\times$ fewer hardware resources than the leading state-of-the-art method. We open-source our framework at https://github.com/ChengZhang-98/lqer

View on arXivView PDF

Code Implementations(1)

ChengZhang-98/LQEROfficial100%

Official implementation of ICML'24 paper "LQER: Low-Rank Quantization Error Reconstruction for LLMs"

192Shell, PythonFeb 15, 20242 years ago

Similar Papers

Low-Rank Correction for Quantized LLMs

90% match2024

ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization

89% match2024

LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices

89% match2024

ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

89% match1 repo43.2k2023

Low-Rank Quantization-Aware Training for LLMs

89% match1 repo542024

L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models

88% match2024

Cite this paper

@article{zhang2024lqer,
  title  = {LQER: Low-Rank Quantization Error Reconstruction for LLMs},
  author = {Cheng Zhang and Jianyi Cheng and George A. Constantinides and Yiren Zhao},
  year   = {2024},
  eprint = {2402.02446},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.02446v3},
  journal = {ICML 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.02446.svg)](https://codeofpaper.com/paper/2402.02446)

Discussion