FreezeOut: Accelerate Training by Progressively Freezing Layers

Andrew Brock, Theodore Lim, J. M. Ritchie, Nick Weston
6/15/2017
stat.MLcs.LG

Official code: ajbrock/FreezeOut (213 stars, PyTorch, last commit 2018-08-15).

Abstract

The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. Through experiments on CIFAR, we empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets, a 20% speedup without loss of accuracy for ResNets, and no improvement for VGG networks. Our code is publicly available at https://github.com/ajbrock/FreezeOut

View on arXivView PDF

Code Implementations(1)

ajbrock/FreezeOutOfficial100%
PyTorch

Accelerate Neural Net Training by Progressively Freezing Layers

21329PythonJun 15, 20178 years ago
deep-learningdensenetmachine-learningmemesneural-networks+3 more

Similar Papers

Rethinking the Potential of Layer Freezing for Efficient DNN Training

84% match2025

MetaGrad: Adaptive Gradient Quantization with Hypernetworks

83% match2023

Decoupled Greedy Learning of CNNs

83% match1 repo322019

LayerDropBack: A Universally Applicable Approach for Accelerating Training of Deep Networks

82% match1 repo22024

Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learning

81% match2022

Scaling Supervised Local Learning with Augmented Auxiliary Networks

81% match1 repo62024

Cite this paper

@article{brock2017freezeout,
  title  = {FreezeOut: Accelerate Training by Progressively Freezing Layers},
  author = {Andrew Brock and Theodore Lim and J. M. Ritchie and Nick Weston},
  year   = {2017},
  eprint = {1706.04983},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1706.04983}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1706.04983.svg)](https://codeofpaper.com/paper/1706.04983)

Discussion