Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization

Aaron Defazio, Samy Jelassi
1/26/2021
cs.LGcs.AImath.OC

Official code: facebookresearch/madgrad (801 stars, MIT, last commit 2025-01-27).

Abstract

We introduce MADGRAD, a novel optimization method in the family of AdaGrad adaptive gradient methods. MADGRAD shows excellent performance on deep learning optimization problems from multiple fields, including classification and image-to-image tasks in vision, and recurrent and bidirectionally-masked models in natural language processing. For each of these tasks, MADGRAD matches or outperforms both SGD and ADAM in test set performance, even on problems for which adaptive methods normally perform poorly.

View on arXivView PDF

Code Implementations(1)

facebookresearch/madgradOfficial100%
MIT

MADGRAD Optimization Method

80156PythonJan 12, 20211 years agoMIT

Similar Papers

AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients

87% match1 repo1.1k2020

Accelerated Gradient Descent for Faster Convergence with Minimal Overhead

86% match2026

MADA: Meta-Adaptive Optimizers through hyper-gradient Descent

86% match1 repo52024

Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates

85% match1 repo32016

Learning with Constraint Learning: New Perspective, Solution Strategy and Various Applications

85% match1 repo132023

Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis

84% match1 repo542025

Cite this paper

@article{defazio2021adaptivity,
  title  = {Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization},
  author = {Aaron Defazio and Samy Jelassi},
  year   = {2021},
  eprint = {2101.11075},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2101.11075}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2101.11075.svg)](https://codeofpaper.com/paper/2101.11075)

Discussion