Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapes

Nikita Kiselev, Andrey Grabovoy
9/18/2024
cs.LG

No official code found. Best implementation: kisnikser/landscape-hessian (3 stars, MIT, last commit 2025-05-16).

Abstract

The loss landscape of neural networks is a critical aspect of their training, and understanding its properties is essential for improving their performance. In this paper, we investigate how the loss surface changes when the sample size increases, a previously unexplored issue. We theoretically analyze the convergence of the loss landscape in a fully connected neural network and derive upper bounds for the difference in loss function values when adding a new object to the sample. Our empirical study confirms these results on various datasets, demonstrating the convergence of the loss function surface for image classification tasks. Our findings provide insights into the local geometry of neural loss landscapes and have implications for the development of sample size determination techniques.

View on arXivView PDF

Code Implementations(1)

kisnikser/landscape-hessianCommunity100%
MIT

Official implementation of "Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapes"

31TeX, PythonAug 10, 20241 years agoMIT
convergence-analysishessian-matriximage-classificationloss-function-landscapeneural-networks

Similar Papers

The instabilities of large learning rate training: a loss landscape view

87% match2023

A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs

86% match2026

Evaluating Loss Landscapes from a Topology Perspective

85% match2024

Investigating generalization capabilities of neural networks by means of loss landscapes and Hessian analysis

85% match1 repo152024

SANE: The phases of gradient descent through Sharpness Adjusted Number of Effective parameters

85% match2023

Super Consistency of Neural Network Landscapes and Learning Rate Transfer

85% match2026

Cite this paper

@article{kiselev2024unraveling,
  title  = {Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapes},
  author = {Nikita Kiselev and Andrey Grabovoy},
  year   = {2024},
  eprint = {2409.11995},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.11995v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.11995.svg)](https://codeofpaper.com/paper/2409.11995)

Discussion