Essentially No Barriers in Neural Network Energy Landscape

Felix Draxler, Kambis Veschgini, Manfred Salmhofer, Fred A. Hamprecht
3/2/2018
stat.MLcs.AIcs.LG

Official code: fdraxler/PyTorch-AutoNEB (57 stars, MIT, PyTorch, last commit 2022-09-14).

Abstract

Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.

View on arXivView PDF

Code Implementations(1)

fdraxler/PyTorch-AutoNEBOfficial100%
PyTorchMIT

PyTorch AutoNEB implementation to identify minimum energy paths, e.g. in neural network loss landscapes

5713Python, Jupyter NotebookJun 6, 20184 years agoMIT
autonebnebneural-networkpythonpytorch

Similar Papers

Visualizing the Loss Landscape of Neural Nets

85% match1 repo3.2k2017

Neural network optimization strategies and the topography of the loss landscape

85% match2026

On the existence of minimizers in shallow residual ReLU neural network optimization landscapes

84% match2023

Flat Channels to Infinity in Neural Loss Landscapes

84% match2025

One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM

84% match2026

Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective

84% match2026

Cite this paper

@article{draxler2018essentially,
  title  = {Essentially No Barriers in Neural Network Energy Landscape},
  author = {Felix Draxler and Kambis Veschgini and Manfred Salmhofer and Fred A. Hamprecht},
  year   = {2018},
  eprint = {1803.00885},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1803.00885}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1803.00885.svg)](https://codeofpaper.com/paper/1803.00885)

Discussion