Exploring loss function topology with cyclical learning rates

Leslie N. Smith, Nicholay Topin
2/14/2017
cs.LGcs.NE

Official code: lnsmith54/exploring-loss (21 stars, GPL-3.0, last commit 2017-03-02).

Abstract

We present observations and discussion of previously unreported phenomena discovered while training residual networks. The goal of this work is to better understand the nature of neural networks through the examination of these new empirical results. These behaviors were identified through the application of Cyclical Learning Rates (CLR) and linear network interpolation. Among these behaviors are counterintuitive increases and decreases in training loss and instances of rapid training. For example, we demonstrate how CLR can produce greater testing accuracy than traditional training despite using large learning rates. Files to replicate these results are available at https://github.com/lnsmith54/exploring-loss

View on arXivView PDF

Code Implementations(1)

lnsmith54/exploring-lossOfficial100%
GPL-3.0
219Shell, PythonFeb 13, 20179 years agoGPL-3.0

Similar Papers

Understanding the Generalization Benefits of Late Learning Rate Decay

86% match2024

Averaging Weights Leads to Wider Optima and Better Generalization

85% match1 repo9832018

Visualizing the Loss Landscape of Neural Nets

84% match1 repo3.2k2017

Loss Spike in Training Neural Networks

84% match2023

The instabilities of large learning rate training: a loss landscape view

83% match2023

LCA: Loss Change Allocation for Neural Network Training

83% match1 repo602019

Cite this paper

@article{smith2017exploring,
  title  = {Exploring loss function topology with cyclical learning rates},
  author = {Leslie N. Smith and Nicholay Topin},
  year   = {2017},
  eprint = {1702.04283},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1702.04283}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1702.04283.svg)](https://codeofpaper.com/paper/1702.04283)

Discussion