Soft Weight-Sharing for Neural Network Compression

Karen Ullrich, Edward Meeds, Max Welling
2/13/2017
stat.MLcs.LG

Official code: KarenUllrich/Tutorial-SoftWeightSharingForNNCompression (147 stars, last commit 2017-02-15).

Abstract

The success of deep learning in numerous application domains created the de- sire to run and train them on mobile devices. This however, conflicts with their computationally, memory and energy intense nature, leading to a growing interest in compression. Recent work by Han et al. (2015a) propose a pipeline that involves retraining, pruning and quantization of neural network weights, obtaining state-of-the-art compression rates. In this paper, we show that competitive compression rates can be achieved by using a version of soft weight-sharing (Nowlan & Hinton, 1992). Our method achieves both quantization and pruning in one simple (re-)training procedure. This point of view also exposes the relation between compression and the minimum description length (MDL) principle.

View on arXivView PDF

Code Implementations(1)

KarenUllrich/Tutorial-SoftWeightSharingForNNCompressionOfficial100%

A tutorial on 'Soft weight-sharing for Neural Network compression' published at ICLR2017

14730Python, Jupyter NotebookFeb 13, 20179 years ago

Similar Papers

Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters

87% match1 repo222018

A Novel Structure-Agnostic Multi-Objective Approach for Weight-Sharing Compression in Deep Neural Networks

86% match2025

Model compression via distillation and quantization

85% match1 repo3352018

JPEG Inspired Deep Learning

85% match1 repo32024

Balanced and Deterministic Weight-sharing Helps Network Performance

85% match2023

Flexible Communication for Optimal Distributed Learning over Unpredictable Networks

84% match1 repo2023

Cite this paper

@article{ullrich2017soft,
  title  = {Soft Weight-Sharing for Neural Network Compression},
  author = {Karen Ullrich and Edward Meeds and Max Welling},
  year   = {2017},
  eprint = {1702.04008},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1702.04008}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1702.04008.svg)](https://codeofpaper.com/paper/1702.04008)

Discussion