In-Depth Look at Word Filling Societal Bias Measures

Matúš Pikuliak, Ivana Beňová, Viktor Bachratý
2/24/2023
cs.CL

No confident code implementation found for this paper.

Abstract

Many measures of societal bias in language models have been proposed in recent years. A popular approach is to use a set of word filling prompts to evaluate the behavior of the language models. In this work, we analyze the validity of two such measures -- StereoSet and CrowS-Pairs. We show that these measures produce unexpected and illogical results when appropriate control group samples are constructed. Based on this, we believe that they are problematic and using them in the future should be reconsidered. We propose a way forward with an improved testing protocol. Finally, we also introduce a new gender bias dataset for Slovak.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples

91% match2023

Measuring Gender Bias in West Slavic Language Models

90% match2023

Debiasing Algorithm through Model Adaptation

89% match1 repo92026

Debiasing Algorithm through Model Adaptation

89% match1 repo92023

On Evaluating and Mitigating Gender Biases in Multilingual Settings

89% match2023

Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implications

89% match2023

Cite this paper

@article{pikuliak2023indepth,
  title  = {In-Depth Look at Word Filling Societal Bias Measures},
  author = {Matúš Pikuliak and Ivana Beňová and Viktor Bachratý},
  year   = {2023},
  eprint = {2302.12640},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.12640v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.12640.svg)](https://codeofpaper.com/paper/2302.12640)

Discussion