GenderBench: Evaluation Suite for Gender Biases in LLMs

Matúš Pikuliak
5/17/2025
cs.CL

No official code found. Best implementation: matus-pikuliak/genderbench (5 stars, last commit 2025-05-17).

Abstract

We present GenderBench -- a comprehensive evaluation suite designed to measure gender biases in LLMs. GenderBench includes 14 probes that quantify 19 gender-related harmful behaviors exhibited by LLMs. We release GenderBench as an open-source and extensible library to improve the reproducibility and robustness of benchmarking across the field. We also publish our evaluation of 12 LLMs. Our measurements reveal consistent patterns in their behavior. We show that LLMs struggle with stereotypical reasoning, equitable gender representation in generated texts, and occasionally also with discriminatory behavior in high-stakes scenarios, such as hiring.

View on arXivView PDF

Code Implementations(1)

matus-pikuliak/genderbenchCommunity100%

GenderBench - Evaluation suite for gender biases in LLMs

51HTML, PythonDec 20, 20241 years agoNOASSERTION
fairness-aigender-biasllmsllms-benchmarking

Similar Papers

Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics

88% match1 repo12024

Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases

87% match1 repo12025

GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models

87% match1 repo272024

Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models

86% match2024

A database to support the evaluation of gender biases in GPT-4o output

85% match2025

Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurement

85% match2025

Cite this paper

@article{pikuliak2025genderbench,
  title  = {GenderBench: Evaluation Suite for Gender Biases in LLMs},
  author = {Matúš Pikuliak},
  year   = {2025},
  eprint = {2505.12054},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.12054v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.12054.svg)](https://codeofpaper.com/paper/2505.12054)

Discussion