The instabilities of large learning rate training: a loss landscape view

Lawrence Wang, Stephen Roberts
7/22/2023
cs.LG

No confident code implementation found for this paper.

Abstract

Modern neural networks are undeniably successful. Numerous works study how the curvature of loss landscapes can affect the quality of solutions. In this work we study the loss landscape by considering the Hessian matrix during network training with large learning rates - an attractive regime that is (in)famously unstable. We characterise the instabilities of gradient descent, and we observe the striking phenomena of \textit{landscape flattening} and \textit{landscape shift}, both of which are intimately connected to the instabilities of training.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Can Stability be Detrimental? Better Generalization through Gradient Descent Instabilities

89% match2024

A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs

88% match2026

SANE: The phases of gradient descent through Sharpness Adjusted Number of Effective parameters

87% match2023

Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapes

87% match1 repo32024

Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignment

86% match2024

Training Instabilities Induce Flatness Bias in Gradient Descent

86% match2025

Cite this paper

@article{wang2023instabilities,
  title  = {The instabilities of large learning rate training: a loss landscape view},
  author = {Lawrence Wang and Stephen Roberts},
  year   = {2023},
  eprint = {2307.11948},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.11948v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.11948.svg)](https://codeofpaper.com/paper/2307.11948)

Discussion