A Novel Interpretability Metric for Explaining Bias in Language Models: Applications on Multilingual Models from Southeast Asia

Lance Calvin Lim Gamboa, Mark Lee
10/20/2024
cs.CL

No official code found. Best implementation: gamboalance/bias_attribution_scores (0 stars, last commit 2025-06-07).

Abstract

Work on bias in pretrained language models (PLMs) focuses on bias evaluation and mitigation and fails to tackle the question of bias attribution and explainability. We propose a novel metric, the $\textit{bias attribution score}$, which draws from information theory to measure token-level contributions to biased behavior in PLMs. We then demonstrate the utility of this metric by applying it on multilingual PLMs, including models from Southeast Asia which have not yet been thoroughly examined in bias evaluation literature. Our results confirm the presence of sexist and homophobic bias in Southeast Asian PLMs. Interpretability and semantic analyses also reveal that PLM bias is strongly induced by words relating to crime, intimate relationships, and helping among other discursive categories, suggesting that these are topics where PLMs strongly reproduce bias from pretraining data and where PLMs should be used with more caution.

View on arXivView PDF

Code Implementations(1)

gamboalance/bias_attribution_scoresCommunity100%

This is the GitHub repository for the paper: A Novel Interpretability Metric for Explaining Bias in Language Models. The paper is part of the 38th Pacific Asia Conference on Language, Information, and Computation (PACLIC 38).

00PythonOct 20, 20241 years ago

Similar Papers

Bias Attribution in Filipino Language Models: Extending a Bias Interpretability Metric for Application on Agglutinative Languages

90% match1 repo2025

Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia

89% match1 repo2024

From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modeling

86% match2025

Bias and Fairness in Large Language Models: A Survey

86% match1 repo1622023

Assessing Bias in Metric Models for LLM Open-Ended Generation Bias Benchmarks

86% match2024

Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective

85% match2025

Cite this paper

@article{gamboa2024novel,
  title  = {A Novel Interpretability Metric for Explaining Bias in Language Models: Applications on Multilingual Models from Southeast Asia},
  author = {Lance Calvin Lim Gamboa and Mark Lee},
  year   = {2024},
  eprint = {2410.15464},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.15464v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.15464.svg)](https://codeofpaper.com/paper/2410.15464)

Discussion