Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent

Benjamin Gess, Sebastian Kassing, Vitalii Konarovskyi
2/14/2023
math.PRcs.LGmath.AP

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified equations by having regular diffusion coefficients and by matching the multi-point statistics. As a second contribution, we introduce distribution dependent stochastic modified flows which we prove to describe the fluctuating limiting dynamics of stochastic gradient descent in the small learning rate - infinite width scaling regime.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Stochastic Modified Flows for Riemannian Stochastic Gradient Descent

88% match2024

High-dimensional learning dynamics of multi-pass Stochastic Gradient Descent in multi-index models

86% match2026

Effective continuous equations for adaptive SGD: a stochastic analysis view

86% match2025

Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features

84% match2026

Is Stochastic Gradient Descent Effective? A PDE Perspective on Machine Learning processes

83% match2025

Effect of Random Learning Rate: Theoretical Analysis of SGD Dynamics in Non-Convex Optimization via Stationary Distribution

82% match2024

Cite this paper

@article{gess2023stochastic,
  title  = {Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent},
  author = {Benjamin Gess and Sebastian Kassing and Vitalii Konarovskyi},
  year   = {2023},
  eprint = {2302.07125},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.07125v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.07125.svg)](https://codeofpaper.com/paper/2302.07125)

Discussion