meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting

Xu Sun, Xuancheng Ren, Shuming Ma, Houfeng Wang
6/19/2017
cs.LGcs.AIcs.CL

Official code: lancopku/meProp (111 stars, last commit 2022-03-29).

Abstract

We propose a simple yet effective technique for neural network learning. The forward propagation is computed as usual. In back propagation, only a small subset of the full gradient is computed to update the model parameters. The gradient vectors are sparsified in such a way that only the top-$k$ elements (in terms of magnitude) are kept. As a result, only $k$ rows or columns (depending on the layout) of the weight matrix are modified, leading to a linear reduction ($k$ divided by the vector dimension) in the computational cost. Surprisingly, experimental results demonstrate that we can update only 1-4% of the weights at each back propagation pass. This does not result in a larger number of training iterations. More interestingly, the accuracy of the resulting models is actually improved rather than degraded, and a detailed analysis is given. The code is available at https://github.com/lancopku/meProp

View on arXivView PDF

Code Implementations(1)

lancopku/mePropOfficial100%

meProp: Sparsified Back Propagation for Accelerated Deep Learning (ICML 2017)

11120C#, PythonAug 27, 20174 years ago
back-propagationcomputer-visiondeep-learningmepropnatural-language-processing+1 more

Similar Papers

Training Simplification and Model Simplification for Deep Learning: A Minimal Effort Back Propagation Method

95% match2 repos1112017

Scaling Private Deep Learning with Low-Rank and Sparse Gradients

86% match2022

Power-Enhanced Residual Network for Function Approximation and Physics-Informed Inverse Problems

86% match2023

A Novel Method for improving accuracy in neural network by reinstating traditional back propagation technique

85% match2023

Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

85% match2026

Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method

85% match1 repo2026

Cite this paper

@article{sun2017meprop,
  title  = {meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting},
  author = {Xu Sun and Xuancheng Ren and Shuming Ma and Houfeng Wang},
  year   = {2017},
  eprint = {1706.06197},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1706.06197}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1706.06197.svg)](https://codeofpaper.com/paper/1706.06197)

Discussion