ASGO: Adaptive Structured Gradient Optimization

Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma +2 more
3/26/2025
cs.LGmath.OC

Official code: infinity-stars/ASGO (11 stars, MIT, last commit 2025-10-29).

Abstract

Training deep neural networks is a structured optimization problem, because the parameters are naturally represented by matrices and tensors rather than by vectors. Under this structural representation, it has been widely observed that gradients are low-rank and Hessians are approximately block diagonal. These structured properties are crucial for designing efficient optimization algorithms, but are not utilized by many current popular optimizers like Adam. In this paper, we present a novel optimization algorithm ASGO that capitalizes on these properties by employing a preconditioner that is adaptively updated using structured gradients. By a fine-grained theoretical analysis, ASGO is proven to achieve superior convergence rates compared to existing structured gradient methods. Based on this convergence theory, we further demonstrate that ASGO can benefit from low-rank gradients and block diagonal Hessians. We also discuss practical modifications of ASGO and empirically verify ASGO's effectiveness on language model tasks. Code is available at https://github.com/infinity-stars/ASGO.

View on arXivView PDF

Code Implementations(1)

infinity-stars/ASGOOfficial100%
MIT

The implementation of optimizer ASGO (https://arxiv.org/abs/2503.20762)

111Shell, PythonOct 24, 202511 months agoMIT

Similar Papers

Towards Guided Descent: Optimization Algorithms for Training Neural Networks At Scale

84% match2025

Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization

84% match2024

PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective

83% match1 repo182025

Dynamic Low-rank Approximation of Full-Matrix Preconditioner for Training Generalized Linear Models

82% match2025

Optimization Methods in Deep Learning: A Comprehensive Overview

81% match2023

Contextual Gradient Flow Modeling for Large Language Model Generalization in Multi-Scale Feature Spaces

81% match2025

Cite this paper

@article{an2025asgo,
  title  = {ASGO: Adaptive Structured Gradient Optimization},
  author = {Kang An and Yuxing Liu and Rui Pan and Yi Ren and Shiqian Ma and Donald Goldfarb and Tong Zhang},
  year   = {2025},
  eprint = {2503.20762},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2503.20762v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2503.20762.svg)](https://codeofpaper.com/paper/2503.20762)

Discussion