Empirical Loss Landscape Analysis of Neural Network Activation Functions

Anna Sergeevna Bosman, Andries Engelbrecht, Marde Helbig
6/28/2023
cs.LGcs.AIcs.NE

Official code: annabosman/fla-in-tf (4 stars, last commit 2020-01-09).

Abstract

Activation functions play a significant role in neural network design by enabling non-linearity. The choice of activation function was previously shown to influence the properties of the resulting loss landscape. Understanding the relationship between activation functions and loss landscape properties is important for neural architecture and training algorithm design. This study empirically investigates neural network loss landscapes associated with hyperbolic tangent, rectified linear unit, and exponential linear unit activation functions. Rectified linear unit is shown to yield the most convex loss landscape, and exponential linear unit is shown to yield the least flat loss landscape, and to exhibit superior generalisation performance. The presence of wide and narrow valleys in the loss landscape is established for all activation functions, and the narrow valleys are shown to correlate with saturated neurons and implicitly regularised network configurations.

View on arXivView PDF

Code Implementations(1)

annabosman/fla-in-tfOfficial100%

Fitness Landscape Analysis metrics implemented in Python/TensorFlow

43Shell, PythonDec 12, 20176 years ago

Similar Papers

DL101 Neural Network Outputs and Loss Functions

84% match2025

A topological description of loss surfaces based on Betti Numbers

83% match2024

Investigating generalization capabilities of neural networks by means of loss landscapes and Hessian analysis

83% match1 repo152024

Dynamical loss functions shape landscape topography and improve learning in artificial neural networks

83% match1 repo32024

The instabilities of large learning rate training: a loss landscape view

82% match2023

Evaluating Loss Landscapes from a Topology Perspective

82% match2024

Cite this paper

@article{bosman2023empirical,
  title  = {Empirical Loss Landscape Analysis of Neural Network Activation Functions},
  author = {Anna Sergeevna Bosman and Andries Engelbrecht and Marde Helbig},
  year   = {2023},
  eprint = {2306.16090},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.16090v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.16090.svg)](https://codeofpaper.com/paper/2306.16090)

Discussion