Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descent

Adrien Schertzer, Loucas Pillaud-Vivien
7/2/2024
cs.LGmath.PR

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

We study the dynamics of a continuous-time model of the Stochastic Gradient Descent (SGD) for the least-square problem. Indeed, pursuing the work of Li et al. (2019), we analyze Stochastic Differential Equations (SDEs) that model SGD either in the case of the training loss (finite samples) or the population one (online setting). A key qualitative feature of the dynamics is the existence of a perfect interpolator of the data, irrespective of the sample size. In both scenarios, we provide precise, non-asymptotic rates of convergence to the (possibly degenerate) stationary distribution. Additionally, we describe this asymptotic distribution, offering estimates of its mean, deviations from it, and a proof of the emergence of heavy-tails related to the step-size magnitude. Numerical simulations supporting our findings are also presented.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Effective continuous equations for adaptive SGD: a stochastic analysis view

84% match2025

Statistical Inference for Linear Functionals of Online Least-squares SGD when $t \gtrsim d^{1+δ}$

83% match2025

Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime

83% match2025

What is the long-run distribution of stochastic gradient descent? A large deviations analysis

83% match2024

High-dimensional learning dynamics of multi-pass Stochastic Gradient Descent in multi-index models

82% match2026

Stochastic Gradient Descent in Non-Convex Problems: Asymptotic Convergence with Relaxed Step-Size via Stopping Time Methods

82% match2025

Cite this paper

@article{schertzer2024stochastic,
  title  = {Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descent},
  author = {Adrien Schertzer and Loucas Pillaud-Vivien},
  year   = {2024},
  eprint = {2407.02322},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.02322v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.02322.svg)](https://codeofpaper.com/paper/2407.02322)

Discussion