The Reversible Residual Network: Backpropagation Without Storing Activations

Aidan N. Gomez, Mengye Ren, Raquel Urtasun, Roger B. Grosse
7/14/2017
cs.CVcs.LG

Official code: renmengye/revnet-public (363 stars, MIT, last commit 2018-06-19).

Abstract

Deep residual networks (ResNets) have significantly pushed forward the state-of-the-art on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck, as one needs to store the activations in order to calculate gradients using backpropagation. We present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backpropagation. We demonstrate the effectiveness of RevNets on CIFAR-10, CIFAR-100, and ImageNet, establishing nearly identical classification accuracy to equally-sized ResNets, even though the activation storage requirements are independent of depth.

View on arXivView PDF

Code Implementations(1)

renmengye/revnet-publicOfficial100%
MIT

Code for "The Reversible Residual Network: Backpropagation Without Storing Activations"

36351Shell, PythonJun 28, 20178 years agoMIT

Similar Papers

Deep Residual Learning for Image Recognition

87% match1 repo9382015

Scaling Supervised Local Learning with Augmented Auxiliary Networks

86% match1 repo62024

ShaResNet: reducing residual network parameter number by sharing weights

86% match1 repo322017

Wide Residual Networks

85% match1 repo1.3k2016

Deep Pyramidal Residual Networks

85% match1 repo1312016

A Partially Reversible U-Net for Memory-Efficient Volumetric Image Segmentation

85% match2 repos1562019

Cite this paper

@article{gomez2017reversible,
  title  = {The Reversible Residual Network: Backpropagation Without Storing Activations},
  author = {Aidan N. Gomez and Mengye Ren and Raquel Urtasun and Roger B. Grosse},
  year   = {2017},
  eprint = {1707.04585},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1707.04585}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1707.04585.svg)](https://codeofpaper.com/paper/1707.04585)

Discussion