Adaptive-saturated RNN: Remember more with less instability

Khoi Minh Nguyen-Duy, Quang Pham, Binh T. Nguyen
4/24/2023
cs.LG

Official code: ndminhkhoi46/asRNN (2 stars, last commit 2023-04-29).

Abstract

Orthogonal parameterization is a compelling solution to the vanishing gradient problem (VGP) in recurrent neural networks (RNNs). With orthogonal parameters and non-saturated activation functions, gradients in such models are constrained to unit norms. On the other hand, although the traditional vanilla RNNs are seen to have higher memory capacity, they suffer from the VGP and perform badly in many applications. This work proposes Adaptive-Saturated RNNs (asRNN), a variant that dynamically adjusts its saturation level between the two mentioned approaches. Consequently, asRNN enjoys both the capacity of a vanilla RNN and the training stability of orthogonal RNNs. Our experiments show encouraging results of asRNN on challenging sequence learning benchmarks compared to several strong competitors. The research code is accessible at https://github.com/ndminhkhoi46/asRNN/.

View on arXivView PDF

Code Implementations(1)

ndminhkhoi46/asRNNOfficial100%

Adaptive-saturated RNN: Remember more with less instability

20PythonJul 25, 20223 years ago
adaptive-saturationmemory-capacityrecurrent-neural-networksvanishing-gradient

Similar Papers

Recurrent neural networks: vanishing and exploding gradients are not the end of the story

84% match1 repo12024

Gradient-Free Training of Recurrent Neural Networks using Random Perturbations

79% match2024

Exploring Learnability in Memory-Augmented Recurrent Neural Networks: Precision, Stability, and Empirical Insights

79% match2024

ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models

79% match2025

Weight-Space Linear Recurrent Neural Networks

79% match1 repo72025

Balancing Sparse RNNs with Hyperparameterization Benefiting Meta-Learning

79% match2025

Cite this paper

@article{nguyenduy2023adaptivesaturated,
  title  = {Adaptive-saturated RNN: Remember more with less instability},
  author = {Khoi Minh Nguyen-Duy and Quang Pham and Binh T. Nguyen},
  year   = {2023},
  eprint = {2304.11790},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.11790v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.11790.svg)](https://codeofpaper.com/paper/2304.11790)

Discussion