Q-Sparse: All Large Language Models can be Fully Sparsely-Activated

Hongyu Wang, Shuming Ma, Ruiping Wang, Furu Wei
7/15/2024
cs.CLcs.LG

No official code found. Best implementation: nanowell/Q-Sparse-LLM (37 stars, MIT, last commit 2024-08-14).

Abstract

We introduce, Q-Sparse, a simple yet effective approach to training sparsely-activated large language models (LLMs). Q-Sparse enables full sparsity of activations in LLMs which can bring significant efficiency gains in inference. This is achieved by applying top-K sparsification to the activations and the straight-through-estimator to the training. We also introduce Block Q-Sparse for batch training and inference. The key results from this work are, (1) Q-Sparse can achieve results comparable to those of baseline LLMs while being much more efficient at inference time; (2) We present an inference-optimal scaling law for sparsely-activated LLMs; (3) Q-Sparse is effective in different settings, including training-from-scratch, continue-training of off-the-shelf LLMs, and finetuning; (4) Q-Sparse works for both full-precision and 1-bit LLMs (e.g., BitNet b1.58). Particularly, the synergy of BitNet b1.58 and Q-Sparse (can be equipped with MoE) provides the cornerstone and a clear path to revolutionize the efficiency, including cost and energy consumption, of future LLMs.

View on arXivView PDF

Code Implementations(1)

nanowell/Q-Sparse-LLMCommunity70%
MIT

My Implementation of Q-Sparse: All Large Language Models can be Fully Sparsely-Activated

371PythonJul 21, 20242 years agoMIT
artificial-intelligencedeep-neural-networksdeeplearninglarge-language-modelsmachine-learning+3 more

Similar Papers

CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models

88% match2025

Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity

87% match1 repo172026

Pruning Large Language Models with Semi-Structural Adaptive Sparse Training

87% match1 repo202024

SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models

87% match2023

Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment

86% match2024

Training-Free Activation Sparsity in Large Language Models

86% match1 repo1722026

Cite this paper

@article{wang2024qsparse,
  title  = {Q-Sparse: All Large Language Models can be Fully Sparsely-Activated},
  author = {Hongyu Wang and Shuming Ma and Ruiping Wang and Furu Wei},
  year   = {2024},
  eprint = {2407.10969},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.10969v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.10969.svg)](https://codeofpaper.com/paper/2407.10969)

Discussion