TEQ: Trainable Equivalent Transformation for Quantization of LLMs

Wenhua Cheng, Yiyang Cai, Kaokao Lv, Haihao Shen
10/17/2023
cs.CL

Official code: intel/neural-compressor (2,672 stars, Apache-2.0, last commit 2026-04-30).

Abstract

As large language models (LLMs) become more prevalent, there is a growing need for new and improved quantization methods that can meet the computationalast layer demands of these modern architectures while maintaining the accuracy. In this paper, we present TEQ, a trainable equivalent transformation that preserves the FP32 precision of the model output while taking advantage of low-precision quantization, especially 3 and 4 bits weight-only quantization. The training process is lightweight, requiring only 1K steps and fewer than 0.1 percent of the original model's trainable parameters. Furthermore, the transformation does not add any computational overhead during inference. Our results are on-par with the state-of-the-art (SOTA) methods on typical LLMs. Our approach can be combined with other methods to achieve even better performance. The code is available at https://github.com/intel/neural-compressor.

View on arXivView PDF

Code Implementations(1)

intel/neural-compressorOfficial100%
Apache-2.0

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

2,672313Roff, ShellJul 21, 20205 months agoApache-2.0
auto-tuningawqfp4gptqint4+12 more

Similar Papers

PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression

89% match2026

CBQ: Cross-Block Quantization for Large Language Models

89% match2023

LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

89% match1 repo3292023

OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models

88% match1 repo9052026

Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs

87% match2024

TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization

87% match1 repo92026

Cite this paper

@article{cheng2023teq,
  title  = {TEQ: Trainable Equivalent Transformation for Quantization of LLMs},
  author = {Wenhua Cheng and Yiyang Cai and Kaokao Lv and Haihao Shen},
  year   = {2023},
  eprint = {2310.10944},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.10944v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.10944.svg)](https://codeofpaper.com/paper/2310.10944)

Discussion