Understanding Variational Autoencoders with Intrinsic Dimension and Information Imbalance

Charles Camboulin, Diego Doimo, Aldo Glielmo
11/4/2024
cs.LGcs.AIstat.ML

No official code found. Best implementation: bancaditalia/Understanding-Variational-Autoencoders-with-Intrinsic-Dimension-and-Information-Imbalance (1 star, AGPL-3.0, last commit 2024-11-04).

Abstract

This work presents an analysis of the hidden representations of Variational Autoencoders (VAEs) using the Intrinsic Dimension (ID) and the Information Imbalance (II). We show that VAEs undergo a transition in behaviour once the bottleneck size is larger than the ID of the data, manifesting in a double hunchback ID profile and a qualitative shift in information processing as captured by the II. Our results also highlight two distinct training phases for architectures with sufficiently large bottleneck sizes, consisting of a rapid fit and a slower generalisation, as assessed by a differentiated behaviour of ID, II, and KL loss. These insights demonstrate that II and ID could be valuable tools for aiding architecture search, for diagnosing underfitting in VAEs, and, more broadly, they contribute to advancing a unified understanding of deep generative models through geometric analysis.

View on arXivView PDF

Code Implementations(1)

bancaditalia/Understanding-Variational-Autoencoders-with-Intrinsic-Dimension-and-Information-ImbalanceCommunity100%
AGPL-3.0

Code to reproduce the results of the paper "Understanding Variational Autoencoders with Intrinsic Dimension and Information Imbalance"

10Python, Jupyter NotebookOct 22, 20241 years agoAGPL-3.0

Similar Papers

Scaling Interpretable Transformers with Parity Bottleneck Layers

82% match2026

Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

82% match1 repo39.0k2020

Are We Using Autoencoders in a Wrong Way?

81% match2023

Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders

81% match2026

DSD$^2$: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?

80% match1 repo32023

Quantifying Overfitting: Introducing the Overfitting Index

79% match2023

Cite this paper

@article{camboulin2024understanding,
  title  = {Understanding Variational Autoencoders with Intrinsic Dimension and Information Imbalance},
  author = {Charles Camboulin and Diego Doimo and Aldo Glielmo},
  year   = {2024},
  eprint = {2411.01978},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.01978v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.01978.svg)](https://codeofpaper.com/paper/2411.01978)

Discussion