DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables

Darshan C. Ganji, Saad Ashfaq, Ehsan Saboori, Sudhakar Sah, Saptarshi Mitra +4 more
4/18/2023
cs.LG

No official code found. Best implementation: Emily0330/DeepGEMM-paper-reproduce (0 stars, last commit 2025-01-07).

Abstract

A lot of recent progress has been made in ultra low-bit quantization, promising significant improvements in latency, memory footprint and energy consumption on edge devices. Quantization methods such as Learned Step Size Quantization can achieve model accuracy that is comparable to full-precision floating-point baselines even with sub-byte quantization. However, it is extremely challenging to deploy these ultra low-bit quantized models on mainstream CPU devices because commodity SIMD (Single Instruction, Multiple Data) hardware typically supports no less than 8-bit precision. To overcome this limitation, we propose DeepGEMM, a lookup table based approach for the execution of ultra low-precision convolutional neural networks on SIMD hardware. The proposed method precomputes all possible products of weights and activations, stores them in a lookup table, and efficiently accesses them at inference time to avoid costly multiply-accumulate operations. Our 2-bit implementation outperforms corresponding 8-bit integer kernels in the QNNPACK framework by up to 1.74x on x86 platforms.

View on arXivView PDF

Code Implementations(1)

Emily0330/DeepGEMM-paper-reproduceCommunity69%

Reproduce the paper (DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables).

00C++Jan 7, 20251 years ago

Similar Papers

LUT-DLA: Lookup Table as Efficient Extreme Low-Bit Deep Learning Accelerator

85% match2025

Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications

84% match2025

Pixel Embedding: Fully Quantized Convolutional Neural Network with Differentiable Lookup Table

84% match2024

Low-bit Model Quantization for Deep Neural Networks: A Survey

83% match1 repo1852025

Gradient-based Automatic Mixed Precision Quantization for Neural Networks On-Chip

83% match2 repos402024

MaRVIn: A Cross-Layer Mixed-Precision RISC-V Framework for DNN Inference, from ISA Extension to Hardware Acceleration

83% match2025

Cite this paper

@article{ganji2023deepgemm,
  title  = {DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables},
  author = {Darshan C. Ganji and Saad Ashfaq and Ehsan Saboori and Sudhakar Sah and Saptarshi Mitra and MohammadHossein AskariHemmat and Alexander Hoffman and Ahmed Hassanien and Mathieu Léonardon},
  year   = {2023},
  eprint = {2304.09049},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.09049v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.09049.svg)](https://codeofpaper.com/paper/2304.09049)

Discussion