The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size

Vardan Papyan
11/16/2018
cs.LGstat.ML

Official code: AnonymousNIPS2019/DeepnetHessian (19 stars, last commit 2019-05-19).

Abstract

We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate efficiently the spectrum of the Hessian of modern deepnets, with tens of millions of parameters, trained on real data. Our results corroborate previous findings, based on small-scale networks, that the Hessian exhibits "spiked" behavior, with several outliers isolated from a continuous bulk. We decompose the Hessian into different components and study the dynamics with training and sample size of each term individually.

View on arXivView PDF

Code Implementations(1)

AnonymousNIPS2019/DeepnetHessianOfficial100%

The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size

192Python, Jupyter NotebookMay 19, 20197 years ago

Similar Papers

Backpropagation-Friendly Eigendecomposition

80% match1 repo722019

A Theory of Saddle Escape in Deep Nonlinear Networks

79% match2026

Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking

79% match2026

LayerDropBack: A Universally Applicable Approach for Accelerating Training of Deep Networks

79% match1 repo22024

Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression

79% match2026

SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning

78% match1 repo1292024

Cite this paper

@article{papyan2018full,
  title  = {The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size},
  author = {Vardan Papyan},
  year   = {2018},
  eprint = {1811.07062},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1811.07062}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1811.07062.svg)](https://codeofpaper.com/paper/1811.07062)

Discussion