Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition

Kenzo Clauw, Sebastiano Stramaglia, Daniele Marinazzo
8/16/2024
cs.LGcs.IT

No confident code implementation found for this paper.

Abstract

This paper studies emergent phenomena in neural networks by focusing on grokking where models suddenly generalize after delayed memorization. To understand this phase transition, we utilize higher-order mutual information to analyze the collective behavior (synergy) and shared properties (redundancy) between neurons during training. We identify distinct phases before grokking allowing us to anticipate when it occurs. We attribute grokking to an emergent phase transition caused by the synergistic interactions between neurons as a whole. We show that weight decay and weight initialization can enhance the emergent phase.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Grokking as a First Order Phase Transition in Two Layer Networks

85% match2026

A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks

85% match1 repo82023

Tracing the Path to Grokking: Embeddings, Dropout, and Network Activation

85% match2025

Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking

84% match1 repo302025

Progress measures for grokking via mechanistic interpretability

83% match2 repos912023

Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok

83% match2023

Cite this paper

@article{clauw2024informationtheoretic,
  title  = {Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition},
  author = {Kenzo Clauw and Sebastiano Stramaglia and Daniele Marinazzo},
  year   = {2024},
  eprint = {2408.08944},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.08944v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.08944.svg)](https://codeofpaper.com/paper/2408.08944)

Discussion