Quasi-hyperbolic momentum and Adam for deep learning

Jerry Ma, Denis Yarats
10/16/2018
cs.LGstat.ML

Official code: facebookresearch/qhoptim (102 stars, MIT, last commit 2020-05-15).

Abstract

Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step. We describe numerous connections to and identities with other algorithms, and we characterize the set of two-state optimization algorithms that QHM can recover. Finally, we propose a QH variant of Adam called QHAdam, and we empirically demonstrate that our algorithms lead to significantly improved training in a variety of settings, including a new state-of-the-art result on WMT16 EN-DE. We hope that these empirical results, combined with the conceptual and practical simplicity of QHM and QHAdam, will spur interest from both practitioners and researchers. Code is immediately available.

View on arXivView PDF

Code Implementations(1)

facebookresearch/qhoptimOfficial100%
MIT

Implementations of quasi-hyperbolic optimization algorithms.

10211Shell, PythonDec 28, 20186 years agoMIT

Similar Papers

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

89% match1 repo8222022

Hybrid Momentum Methods for Optimizing Deep Learning Models

85% match2026

Towards understanding how momentum improves generalization in deep learning

85% match2022

On the insufficiency of existing momentum schemes for Stochastic Optimization

85% match1 repo2162018

Scaling transition from momentum stochastic gradient descent to plain stochastic gradient descent

85% match2021

On the Performance Analysis of Momentum Method: A Frequency Domain Perspective

84% match1 repo102024

Cite this paper

@article{ma2018quasihyperbolic,
  title  = {Quasi-hyperbolic momentum and Adam for deep learning},
  author = {Jerry Ma and Denis Yarats},
  year   = {2018},
  eprint = {1810.06801},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1810.06801}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1810.06801.svg)](https://codeofpaper.com/paper/1810.06801)

Discussion