Understanding Imbalanced Semantic Segmentation Through Neural Collapse

Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi +2 more
1/3/2023
cs.CVcs.LG

No official code found. Best implementation: Pointcept/Pointcept (3,150 stars, MIT, PyTorch, last commit 2026-04-07). 2 more below.

Abstract

A recent study has shown a phenomenon called neural collapse in that the within-class means of features and the classifier weight vectors converge to the vertices of a simplex equiangular tight frame at the terminal phase of training for classification. In this paper, we explore the corresponding structures of the last-layer feature centers and classifiers in semantic segmentation. Based on our empirical and theoretical analysis, we point out that semantic segmentation naturally brings contextual correlation and imbalanced distribution among classes, which breaks the equiangular and maximally separated structure of neural collapse for both feature centers and classifiers. However, such a symmetric structure is beneficial to discrimination for the minor classes. To preserve these advantages, we introduce a regularizer on feature centers to encourage the network to learn features closer to the appealing structure in imbalanced semantic segmentation. Experimental results show that our method can bring significant improvements on both 2D and 3D semantic segmentation benchmarks. Moreover, our method ranks 1st and sets a new record (+6.8% mIoU) on the ScanNet200 test leaderboard. Code will be available at https://github.com/dvlab-research/Imbalanced-Learning.

View on arXivView PDF

Code Implementations(3)

Pointcept/PointceptCommunity100%
PyTorchMIT

Pointcept: Perceive the world with sparse points, a codebase for point cloud perception research. Latest works: Utonia, Concerto (NeurIPS'25), Sonata (CVPR'25 Highlight), PTv3 (CVPR'24 Oral)

3,150402C++, CudaMar 21, 20236 months agoMIT
3d-visionpoint-cloudpytorch
NeuralCollapseApplications/Semantic-SegmentationCommunity67%

[CVPR 2023] The official code for our CVPR 2023 paper "Understanding Imbalanced Semantic Segmentation Through Neural Collapse".

80Feb 28, 20233 years ago
imbalanced-learningneural-collapsesemantic-segmentation
akashrma/CeCoCommunity57%
MIT

Unofficial implementation of Understanding Imbalanced Semantic Segmentation Through Neural Collapse (CVPR 2023)

00PythonApr 30, 20251 years agoMIT

Similar Papers

All-around Neural Collapse for Imbalanced Classification

86% match2024

Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data

85% match1 repo102023

Inducing Neural Collapse in Deep Long-tailed Learning

84% match2023

Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model

83% match2024

Generalized Neural Collapse for a Large Number of Classes

83% match2026

Linguistic Collapse: Neural Collapse in (Large) Language Models

83% match2 repos182026

Cite this paper

@article{zhong2023understanding,
  title  = {Understanding Imbalanced Semantic Segmentation Through Neural Collapse},
  author = {Zhisheng Zhong and Jiequan Cui and Yibo Yang and Xiaoyang Wu and Xiaojuan Qi and Xiangyu Zhang and Jiaya Jia},
  year   = {2023},
  eprint = {2301.01100},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2301.01100v1},
  journal = {CVPR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2301.01100.svg)](https://codeofpaper.com/paper/2301.01100)

Discussion