A Simple and Effective Pruning Approach for Large Language Models

Mingjie Sun, Zhuang Liu, Anna Bair, J. Zico Kolter
6/20/2023
cs.CLcs.AIcs.LG

Official code: locuslab/wanda (867 stars, MIT, last commit 2024-08-09). 2 more implementations below.

Abstract

As their size increases, Large Languages Models (LLMs) are natural candidates for network pruning methods: approaches that drop a subset of network weights while striving to preserve performance. Existing methods, however, require either retraining, which is rarely affordable for billion-scale LLMs, or solving a weight reconstruction problem reliant on second-order information, which may also be computationally expensive. In this paper, we introduce a novel, straightforward yet effective pruning method, termed Wanda (Pruning by Weights and activations), designed to induce sparsity in pretrained LLMs. Motivated by the recent observation of emergent large magnitude features in LLMs, our approach prunes weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis. Notably, Wanda requires no retraining or weight update, and the pruned LLM can be used as is. We conduct a thorough evaluation of our method Wanda on LLaMA and LLaMA-2 across various language benchmarks. Wanda significantly outperforms the established baseline of magnitude pruning and performs competitively against recent method involving intensive weight update. Code is available at https://github.com/locuslab/wanda.

View on arXivView PDF

Code Implementations(3)

locuslab/wandaOfficial100%
MIT

A simple and effective LLM pruning approach.

867130Shell, PythonJun 6, 20232 years agoMIT
large-language-modelsnetwork-pruning
tweizy/wanda-prunning-experimentCommunity68%

Implementation of the Wanda prunning technique on LLaMA-2-7B and LLaMA-3.1-8B from the paper: A SIMPLE AND EFFECTIVE PRUNING APPROACH FOR LARGE LANGUAGE MODELS

11Jupyter NotebookJan 10, 20269 months ago
MMiletic03/CAP6614-Course-ProjectCommunity57%

Course project for the course CAP6614 - Current Topics in Machine Learning at the University of Central Florida. The project aims to reproduce the results of the paper 'A Simple and Effective Pruning Approach for Large Language Models' (https://arxiv.org/abs/2306.11695) and additionally perform sensitivity analysis using Meta's LlaMa-2-7B model.

00PythonMar 7, 20266 months ago

Similar Papers

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

92% match1 repo482023

Efficient Post-Training Pruning of Large Language Models with Statistical Correction

90% match2026

Z-Pruner: Post-Training Pruning of Large Language Models for Efficiency without Retraining

90% match1 repo82025

2SSP: A Two-Stage Framework for Structured Pruning of LLMs

89% match1 repo222025

Towards Extreme Pruning of LLMs with Plug-and-Play Mixed Sparsity

89% match2025

Frustratingly Easy Task-aware Pruning for Large Language Models

89% match2025

Cite this paper

@article{sun2023simple,
  title  = {A Simple and Effective Pruning Approach for Large Language Models},
  author = {Mingjie Sun and Zhuang Liu and Anna Bair and J. Zico Kolter},
  year   = {2023},
  eprint = {2306.11695},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.11695v3},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.11695.svg)](https://codeofpaper.com/paper/2306.11695)

Discussion