Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics

Tetiana Bas
11/20/2024
cs.CLcs.LG

No official code found. Best implementation: tanya-bas/LLM_biases (1 star, last commit 2024-10-07).

Abstract

This study investigates gender bias in large language models (LLMs) by comparing their gender perception to that of human respondents, U.S. Bureau of Labor Statistics data, and a 50% no-bias benchmark. We created a new evaluation set using occupational data and role-specific sentences. Unlike common benchmarks included in LLM training data, our set is newly developed, preventing data leakage and test set contamination. Five LLMs were tested to predict the gender for each role using single-word answers. We used Kullback-Leibler (KL) divergence to compare model outputs with human perceptions, statistical data, and the 50% neutrality benchmark. All LLMs showed significant deviation from gender neutrality and aligned more with statistical data, still reflecting inherent biases.

View on arXivView PDF

Code Implementations(1)

tanya-bas/LLM_biasesCommunity100%

A framework for generating new benchmarking datasets anchoring them in the social science studies. Gender bias evaluation as a proof of concept of the technology

10Jupyter NotebookAug 26, 20242 years ago

Similar Papers

Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases

90% match1 repo12025

Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models

90% match2024

Exploring Gender Bias in Large Language Models: An In-depth Dive into the German Language

89% match2025

Gender Bias in LLM-generated Interview Responses

89% match2024

More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models

89% match2025

GenderBench: Evaluation Suite for Gender Biases in LLMs

88% match1 repo52025

Cite this paper

@article{bas2024assessing,
  title  = {Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics},
  author = {Tetiana Bas},
  year   = {2024},
  eprint = {2411.13738},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.13738v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.13738.svg)](https://codeofpaper.com/paper/2411.13738)

Discussion