Feature Collapse

Thomas Laurent, James H. von Brecht, Xavier Bresson
5/25/2023
cs.LGcs.AIcs.CL

No official code found. Best implementation: xbresson/feature_collapse (6 stars, MIT, last commit 2023-05-27).

Abstract

We formalize and study a phenomenon called feature collapse that makes precise the intuitive idea that entities playing a similar role in a learning task receive similar representations. As feature collapse requires a notion of task, we leverage a simple but prototypical NLP task to study it. We start by showing experimentally that feature collapse goes hand in hand with generalization. We then prove that, in the large sample limit, distinct words that play identical roles in this NLP task receive identical local feature representations in a neural network. This analysis reveals the crucial role that normalization mechanisms, such as LayerNorm, play in feature collapse and in generalization.

View on arXivView PDF

Code Implementations(1)

xbresson/feature_collapseCommunity100%
MIT

Code for "Feature Collapse"

60Jupyter NotebookMay 5, 20233 years agoMIT

Similar Papers

Generalized Neural Collapse for a Large Number of Classes

86% match2026

Linguistic Collapse: Neural Collapse in (Large) Language Models

85% match2 repos182026

Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking

83% match2 repos12025

Deep Neural Regression Collapse

83% match1 repo2026

Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay

82% match2023

Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations

82% match2023

Cite this paper

@article{laurent2023feature,
  title  = {Feature Collapse},
  author = {Thomas Laurent and James H. von Brecht and Xavier Bresson},
  year   = {2023},
  eprint = {2305.16162},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.16162v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.16162.svg)](https://codeofpaper.com/paper/2305.16162)

Discussion