Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling

Matúš Pikuliak, Andrea Hrckova, Stefan Oresko, Marián Šimko
11/30/2023
cs.CL

No official code found. Best implementation: kinit-sk/gest (0 stars, Apache-2.0, last commit 2025-02-15).

Abstract

We present GEST -- a new manually created dataset designed to measure gender-stereotypical reasoning in language models and machine translation systems. GEST contains samples for 16 gender stereotypes about men and women (e.g., Women are beautiful, Men are leaders) that are compatible with the English language and 9 Slavic languages. The definition of said stereotypes was informed by gender experts. We used GEST to evaluate English and Slavic masked LMs, English generative LMs, and machine translation systems. We discovered significant and consistent amounts of gender-stereotypical reasoning in almost all the evaluated models and languages. Our experiments confirm the previously postulated hypothesis that the larger the model, the more stereotypical it usually is.

View on arXivView PDF

Code Implementations(1)

kinit-sk/gestCommunity100%
Apache-2.0

GEST - A dataset for measuring gender-stereotypical reasoning in machine translation and language models.

00Python, DockerfileNov 28, 20231 years agoApache-2.0

Similar Papers

EuroGEST: Investigating gender stereotypes in multilingual language models

90% match2025

Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translation

89% match2023

Gender Bias in MT for a Genderless Language: New Benchmarks for Basque

89% match2026

The Gender-GAP Pipeline: A Gender-Aware Polyglot Pipeline for Gender Characterisation in 55 Languages

88% match1 repo2132023

Don't Overlook the Grammatical Gender: Bias Evaluation for Hindi-English Machine Translation

88% match2023

Will the Prince Get True Love's Kiss? On the Model Sensitivity to Gender Perturbation over Fairytale Texts

88% match2023

Cite this paper

@article{pikuliak2023women,
  title  = {Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling},
  author = {Matúš Pikuliak and Andrea Hrckova and Stefan Oresko and Marián Šimko},
  year   = {2023},
  eprint = {2311.18711},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.18711v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.18711.svg)](https://codeofpaper.com/paper/2311.18711)

Discussion