Closing the Curvature Gap: Full Transformer Hessians and Their Implications for Scaling Laws

Egor Petrov, Nikita Kiselev, Vladislav Meshkov, Andrey Grabovoy
10/19/2025
cs.LG

No official code found. Best implementation: modernTalker/Closing-the-Curvature-Gap (0 stars, MIT, last commit 2025-12-20).

Abstract

The lack of theoretical results for Layer Normalization and feedforward Hessians has left a gap in the study of Transformer optimization landscapes. We address this by deriving explicit second-order expressions for these components, thereby completing the Hessian characterization of full Transformer blocks. Our results generalize prior self-attention analyses and yield estimations for the role of each sublayer in curvature propagation. We demonstrate how these Hessian structures inform both convergence dynamics and the empirical scaling laws governing large-model performance. Further, we propose a Taylor-expansion-based framework for analyzing loss differences to quantify convergence trajectories. By extending Hessian theory to the full Transformer architecture, this work establishes a new foundation for theoretical and empirical investigations of optimization in large-scale deep learning.

View on arXivView PDF

Code Implementations(1)

modernTalker/Closing-the-Curvature-GapCommunity75%
MIT

Official paper "Closing the Curvature Gap: Full Transformer Hessians and Their Implications for Scaling Laws" github

00TeXMar 5, 202510 months agoMIT
research-paper

Similar Papers

What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis

86% match1 repo22026

Unraveling the Gradient Descent Dynamics of Transformers

85% match2024

Exact Attention Sensitivity and the Geometry of Transformer Stability

83% match2026

On the Convergence of Gradient Descent on Learning Transformers with Residual Connections

82% match2025

The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit

82% match2023

Geometric and Dynamic Scaling in Deep Transformers

82% match2026

Cite this paper

@article{petrov2025closing,
  title  = {Closing the Curvature Gap: Full Transformer Hessians and Their Implications for Scaling Laws},
  author = {Egor Petrov and Nikita Kiselev and Vladislav Meshkov and Andrey Grabovoy},
  year   = {2025},
  eprint = {2510.16927},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.16927v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.16927.svg)](https://codeofpaper.com/paper/2510.16927)

Discussion