Accelerating Transformer Inference and Training with 2:4 Activation Sparsity

Daniel Haziza, Timothy Chou, Dhruv Choudhary, Luca Wehrstedt, Francisco Massa +5 more
3/20/2025
cs.LGcs.AI

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

In this paper, we demonstrate how to leverage 2:4 sparsity, a popular hardware-accelerated GPU sparsity pattern, to activations to accelerate large language model training and inference. Crucially we exploit the intrinsic sparsity found in Squared-ReLU activations to provide this acceleration with no accuracy loss. Our approach achieves up to 1.3x faster Feed Forward Network (FFNs) in both the forwards and backwards pass. This work highlights the potential for sparsity to play a key role in accelerating large language model training and inference.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training

92% match2026

Mixed Sparsity Training: Achieving 4$\times$ FLOP Reduction for Transformer Pretraining

89% match2024

Beyond 2:4: exploring V:N:M sparsity for efficient transformer inference on GPUs

88% match2024

Learn To be Efficient: Build Structured Sparsity in Large Language Models

88% match2024

Learn To be Efficient: Build Structured Sparsity in Large Language Models

88% match2026

Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs

87% match1 repo82025

Cite this paper

@article{haziza2025accelerating,
  title  = {Accelerating Transformer Inference and Training with 2:4 Activation Sparsity},
  author = {Daniel Haziza and Timothy Chou and Dhruv Choudhary and Luca Wehrstedt and Francisco Massa and Jiecao Yu and Geonhwa Jeong and Supriya Rao and Patrick Labatut and Jesse Cai},
  year   = {2025},
  eprint = {2503.16672},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2503.16672v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2503.16672.svg)](https://codeofpaper.com/paper/2503.16672)

Discussion