Data Caricatures: On the Representation of African American Language in Pretraining Corpora

Nicholas Deas, Blake Vente, Amith Ananthram, Jessica A. Grieser, Desmond Patton +3 more
3/13/2025
cs.CL

No official code found. Best implementation: NickDeas/DataCaricatures (0 stars, last commit 2025-08-20).

Abstract

With a combination of quantitative experiments, human judgments, and qualitative analyses, we evaluate the quantity and quality of African American Language (AAL) representation in 12 predominantly English, open-source pretraining corpora. We specifically focus on the sources, variation, and naturalness of included AAL texts representing the AAL-speaking community. We find that AAL is underrepresented in all evaluated pretraining corpora compared to US demographics, constituting as few as 0.007% and at most 0.18% of documents. We also find that more than 25% of AAL texts in C4 may be perceived as inappropriate for LLMs to generate and to reinforce harmful stereotypes. Finally, we find that most automated filters are more likely to conserve White Mainstream English (WME) texts over AAL in pretraining corpora.

View on arXivView PDF

Code Implementations(1)

NickDeas/DataCaricaturesCommunity60%

Code accompanying "Data Caricatures: On the Representation of African American Language in Pretraining Corpora"

00May 31, 20251 years ago

Similar Papers

Evaluating the Usage of African-American Vernacular English in Large Language Models

87% match2026

My LLM might Mimic AAE -- But When Should it?

85% match1 repo12025

Rejected Dialects: Biases Against African American Language in Reward Models

83% match1 repo2025

Dialect prejudice predicts AI decisions about people's character, employability, and criminality

82% match1 repo382024

Creating an African American-Sounding TTS: Guidelines, Technical Challenges,and Surprising Evaluations

82% match2024

Analysis of LLM as a grammatical feature tagger for African American English

81% match2025

Cite this paper

@article{deas2025data,
  title  = {Data Caricatures: On the Representation of African American Language in Pretraining Corpora},
  author = {Nicholas Deas and Blake Vente and Amith Ananthram and Jessica A. Grieser and Desmond Patton and Shana Kleiner and James Shepard and Kathleen McKeown},
  year   = {2025},
  eprint = {2503.10789},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2503.10789v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2503.10789.svg)](https://codeofpaper.com/paper/2503.10789)

Discussion