Exploding and vanishing gradients in deep neural networks: the effect of residual connections

Vivek S Borkar
6/15/2026
math.OCcs.LG

No confident code implementation found for this paper.

Abstract

The well known phenomenon of exploding and vanishing gradients in deep neural networks is analyzed using multiplicative ergodic theory. The effect of adding a residual connection is explained in this context. Specifically, a characterization of Liapunov exponents due to Furstenberg and Kifer is exploited in order to make a precise statement about the Liapunov spectrum and the effect of residual connections on it.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Gradient Residual Connections

82% match2026

Fading memory as inductive bias in residual recurrent networks

81% match2023

Noise Balance and Stationary Distribution of Stochastic Gradient Descent

81% match1 repo2023

Intelligent gradient amplification for deep neural networks

81% match2023

Stabilizing RNN Gradients through Pre-training

81% match2023

Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization

80% match1 repo242022

Cite this paper

@article{borkar2026exploding,
  title  = {Exploding and vanishing gradients in deep neural networks: the effect of residual connections},
  author = {Vivek S Borkar},
  year   = {2026},
  eprint = {2606.17013},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2606.17013v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2606.17013.svg)](https://codeofpaper.com/paper/2606.17013)

Discussion