A Decade's Battle on Dataset Bias: Are We There Yet?

Zhuang Liu, Kaiming He
3/13/2024
cs.CVcs.LG

No official code found. Best implementation: liuzhuang13/bias (115 stars, MIT, last commit 2024-03-14). 1 more below.

Abstract

We revisit the "dataset classification" experiment suggested by Torralba & Efros (2011) a decade ago, in the new era with large-scale, diverse, and hopefully less biased datasets as well as more capable neural network architectures. Surprisingly, we observe that modern neural networks can achieve excellent accuracy in classifying which dataset an image is from: e.g., we report 84.7% accuracy on held-out validation data for the three-way classification problem consisting of the YFCC, CC, and DataComp datasets. Our further experiments show that such a dataset classifier could learn semantic features that are generalizable and transferable, which cannot be explained by memorization. We hope our discovery will inspire the community to rethink issues involving dataset bias.

View on arXivView PDF

Code Implementations(2)

liuzhuang13/biasCommunity100%
MIT
1151Mar 14, 20242 years agoMIT
Axionis47/dataset-bias-reproductionCommunity64%

Reproduction of 'A Decade's Battle on Dataset Bias: Are We There Yet?' paper with laptop-friendly scaling for Mac M4

10Python, MakefileSep 6, 20251 years ago

Similar Papers

ImageNot: A contrast with ImageNet preserves model rankings

82% match1 repo22024

Identifying Bias in Deep Neural Networks Using Image Transforms

81% match1 repo2024

Mitigating Algorithmic Bias in Multiclass CNN Classifications Using Causal Modeling

80% match2025

Understanding Bias in Large-Scale Visual Datasets

80% match1 repo252024

A Large-Scale Empirical Study on Improving the Fairness of Image Classification Models

80% match1 repo32024

Harnessing Synthetic Datasets: The Role of Shape Bias in Deep Neural Network Generalization

80% match1 repo2023

Cite this paper

@article{liu2024decades,
  title  = {A Decade's Battle on Dataset Bias: Are We There Yet?},
  author = {Zhuang Liu and Kaiming He},
  year   = {2024},
  eprint = {2403.08632},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2403.08632v2},
  journal = {ICLR 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.08632.svg)](https://codeofpaper.com/paper/2403.08632)

Discussion