TP-Aware Dequantization

Adnan Hoque, Mudhakar Srivatsa, Chih-Chieh Yang, Raghu Ganti
1/15/2024
cs.DCcs.LG

No confident code implementation found for this paper.

Abstract

In this paper, we present a novel method that reduces model inference latency during distributed deployment of Large Language Models (LLMs). Our contribution is an optimized inference deployment scheme that address the current limitations of state-of-the-art quantization kernels when used in conjunction with Tensor Parallel (TP). Our method preserves data locality in GPU memory access patterns and exploits a priori knowledge of TP to reduce global communication. We demonstrate an up to 1.81x speedup over existing methods for Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Towards Low-bit Communication for Tensor Parallel LLM Inference

87% match2024

Communication Compression for Tensor Parallel LLM Inference

87% match2024

FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization

85% match2024

LLM Inference Beyond a Single Node: From Bottlenecks to Mitigations with Fast All-Reduce Communication

85% match2025

Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization

85% match1 repo342024

SPD: Sync-Point Drop for efficient tensor parallelism of Large Language Models

85% match2026

Cite this paper

@article{hoque2024tpaware,
  title  = {TP-Aware Dequantization},
  author = {Adnan Hoque and Mudhakar Srivatsa and Chih-Chieh Yang and Raghu Ganti},
  year   = {2024},
  eprint = {2402.04925},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.04925v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.04925.svg)](https://codeofpaper.com/paper/2402.04925)

Discussion