ASDL: A Unified Interface for Gradient Preconditioning in PyTorch

Kazuki Osawa, Satoki Ishikawa, Rio Yokota, Shigang Li, Torsten Hoefler
5/8/2023
cs.LG

No official code found. Best implementation: kazukiosawa/asdl (192 stars, MIT, last commit 2024-12-05).

Abstract

Gradient preconditioning is a key technique to integrate the second-order information into gradients for improving and extending gradient-based learning algorithms. In deep learning, stochasticity, nonconvexity, and high dimensionality lead to a wide variety of gradient preconditioning methods, with implementation complexity and inconsistent performance and feasibility. We propose the Automatic Second-order Differentiation Library (ASDL), an extension library for PyTorch, which offers various implementations and a plug-and-play unified interface for gradient preconditioning. ASDL enables the study and structured comparison of a range of gradient preconditioning methods.

View on arXivView PDF

Code Implementations(1)

kazukiosawa/asdlCommunity100%
MIT

ASDL: Automatic Second-order Differentiation Library for PyTorch

19218PythonSep 3, 20201 years agoMIT

Cite this paper

@article{osawa2023asdl,
  title  = {ASDL: A Unified Interface for Gradient Preconditioning in PyTorch},
  author = {Kazuki Osawa and Satoki Ishikawa and Rio Yokota and Shigang Li and Torsten Hoefler},
  year   = {2023},
  eprint = {2305.04684},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.04684v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.04684.svg)](https://codeofpaper.com/paper/2305.04684)

Discussion