Why Gradients Rapidly Increase Near the End of Training

Aaron Defazio
6/2/2025
cs.LGcs.AI

No confident code implementation found for this paper.

Abstract

During long-duration Large Language Model (LLM) training runs the gradient norm increases rapidly near the end of training. In this short note, we show that this increase is due to an unintended interaction between weight decay, normalization layers, and the learning rate schedule. We propose a simple correction that fixes this behavior while also resulting in lower loss values throughout training.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Methods of improving LLM training stability

84% match2024

Universal One-third Time Scaling in Learning Peaked Distributions

84% match1 repo82026

The Fine-Grained Complexity of Gradient Computation for Training Large Language Models

84% match2026

Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules

83% match2025

The Fair Language Model Paradox

83% match2024

Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning

83% match1 repo72025

Cite this paper

@article{defazio2025gradients,
  title  = {Why Gradients Rapidly Increase Near the End of Training},
  author = {Aaron Defazio},
  year   = {2025},
  eprint = {2506.02285},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.02285v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.02285.svg)](https://codeofpaper.com/paper/2506.02285)

Discussion