Global Convergence of SGD For Logistic Loss on Two Layer Neural Nets

Pulkit Gopalani, Samyak Jha, Anirbit Mukherjee
9/17/2023
cs.LGmath.OCstat.ML

No confident code implementation found for this paper.

Abstract

In this note, we demonstrate a first-of-its-kind provable convergence of SGD to the global minima of appropriately regularized logistic empirical risk of depth $2$ nets -- for arbitrary data and with any number of gates with adequately smooth and bounded activations like sigmoid and tanh. We also prove an exponentially fast convergence rate for continuous time SGD that also applies to smooth unbounded activations like SoftPlus. Our key idea is to show the existence of Frobenius norm regularized logistic loss functions on constant-sized neural nets which are "Villani functions" and thus be able to build on recent progress with analyzing SGD on such objectives.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Fast Convergence in Learning Two-Layer Neural Networks with Separable Data

84% match2023

Langevin Monte-Carlo Provably Learns Depth Two Neural Nets at Any Size and Data

81% match2025

Classification with Deep Neural Networks and Logistic Loss

81% match2023

The Implicit Bias of Gradient Descent on Separable Data

80% match2017

Universal Consistency of Wide and Deep ReLU Neural Networks and Minimax Optimal Convergence Rates for Kolmogorov-Donoho Optimal Function Classes

79% match2024

Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD

79% match2023

Cite this paper

@article{gopalani2023global,
  title  = {Global Convergence of SGD For Logistic Loss on Two Layer Neural Nets},
  author = {Pulkit Gopalani and Samyak Jha and Anirbit Mukherjee},
  year   = {2023},
  eprint = {2309.09258},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.09258v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.09258.svg)](https://codeofpaper.com/paper/2309.09258)

Discussion