QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference

Taesu Kim, Jongho Lee, Daehyun Ahn, Sarang Kim, Jiwoong Choi +2 more
2/15/2024
cs.LGcs.AIcs.CL

No official code found. Best implementation: SqueezeBits/QUICK (123 stars, MIT, last commit 2024-03-06).

Abstract

We introduce QUICK, a group of novel optimized CUDA kernels for the efficient inference of quantized Large Language Models (LLMs). QUICK addresses the shared memory bank-conflict problem of state-of-the-art mixed precision matrix multiplication kernels. Our method interleaves the quantized weight matrices of LLMs offline to skip the shared memory write-back after the dequantization. We demonstrate up to 1.91x speedup over existing kernels of AutoAWQ on larger batches and up to 1.94x throughput gain on representative LLM models on various NVIDIA GPU devices.

View on arXivView PDF

Code Implementations(1)

SqueezeBits/QUICKCommunity100%
MIT

QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference

1235C, C++Feb 15, 20242 years agoMIT

Similar Papers

Fast Matrix Multiplications for Lookup Table-Quantized LLMs

86% match1 repo3892024

LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference

86% match1 repo262024

FPGA Co-Design for Efficient N:M Sparse and Quantized Model Inference

85% match2025

Progressive Mixed-Precision Decoding for Efficient LLM Inference

84% match1 repo2024

Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs

84% match1 repo131.9k2024

A Speed Odyssey for Deployable Quantization of LLMs

84% match2023

Cite this paper

@article{kim2024quick,
  title  = {QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference},
  author = {Taesu Kim and Jongho Lee and Daehyun Ahn and Sarang Kim and Jiwoong Choi and Minkyu Kim and Hyungjun Kim},
  year   = {2024},
  eprint = {2402.10076},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.10076v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.10076.svg)](https://codeofpaper.com/paper/2402.10076)

Discussion