A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'

Ali Borji
10/16/2024
cs.LGcs.AI

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

The study conducted by Shumailov et al. (2024) demonstrates that repeatedly training a generative model on synthetic data leads to model collapse. This finding has generated considerable interest and debate, particularly given that current models have nearly exhausted the available data. In this work, we investigate the effects of fitting a distribution (through Kernel Density Estimation, or KDE) or a model to the data, followed by repeated sampling from it. Our objective is to develop a theoretical understanding of the phenomenon observed by Shumailov et al. (2024). Our results indicate that the outcomes reported are a statistical phenomenon and may be unavoidable.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse

87% match2024

Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

86% match2024

A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective

85% match1 repo52025

When Models Don't Collapse: On the Consistency of Iterative MLE

85% match2025

Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training

85% match2026

Rate of Model Collapse in Recursive Training

83% match1 repo32024

Cite this paper

@article{borji2024note,
  title  = {A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'},
  author = {Ali Borji},
  year   = {2024},
  eprint = {2410.12954},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.12954v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.12954.svg)](https://codeofpaper.com/paper/2410.12954)

Discussion