Criticality versus uniformity in deep neural networks

Aleksandar Bukva, Jurriaan de Gier, Kevin T. Grosvenor, Ro Jefferson, Koenraad Schalm +1 more
4/10/2023
cs.LGstat.ML

No confident code implementation found for this paper.

Abstract

Deep feedforward networks initialized along the edge of chaos exhibit exponentially superior training ability as quantified by maximum trainable depth. In this work, we explore the effect of saturation of the tanh activation function along the edge of chaos. In particular, we determine the line of uniformity in phase space along which the post-activation distribution has maximum entropy. This line intersects the edge of chaos, and indicates the regime beyond which saturation of the activation function begins to impede training efficiency. Our results suggest that initialization along the edge of chaos is a necessary but not sufficient condition for optimal trainability.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks

84% match2023

Scaling and Resizing Symmetry in Feedforward Networks

84% match2023

On the Initialisation of Wide Low-Rank Feedforward Neural Networks

84% match2023

Feature Learning and Generalization in Deep Networks with Orthogonal Weights

83% match2023

Opening the Black Box: predicting the trainability of deep neural networks with reconstruction entropy

82% match1 repo52024

Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width

82% match2023

Cite this paper

@article{bukva2023criticality,
  title  = {Criticality versus uniformity in deep neural networks},
  author = {Aleksandar Bukva and Jurriaan de Gier and Kevin T. Grosvenor and Ro Jefferson and Koenraad Schalm and Eliot Schwander},
  year   = {2023},
  eprint = {2304.04784},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.04784v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.04784.svg)](https://codeofpaper.com/paper/2304.04784)

Discussion