Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models

Yi-Cheng Lin, Wei-Chih Chen, Hung-yi Lee
8/14/2024
cs.CLeess.AS

No official code found. Best implementation: dlion168/spoken_stereoset (1 star, last commit 2025-05-16).

Abstract

Warning: This paper may contain texts with uncomfortable content. Large Language Models (LLMs) have achieved remarkable performance in various tasks, including those involving multimodal data like speech. However, these models often exhibit biases due to the nature of their training data. Recently, more Speech Large Language Models (SLLMs) have emerged, underscoring the urgent need to address these biases. This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in SLLMs. By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. Our experiments reveal significant insights into their performance and bias levels. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.

View on arXivView PDF

Code Implementations(1)

dlion168/spoken_stereosetCommunity100%

The official repo for speech based stereoset

10May 16, 20241 years ago

Similar Papers

Measuring Stereotype and Deviation Biases in Large Language Models

88% match2025

Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Models

86% match1 repo12024

Social Bias Evaluation for Large Language Models Requires Prompt Variations

86% match1 repo2024

Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans

85% match2024

Addressing Stereotypes in Large Language Models: A Critical Examination and Mitigation

85% match2025

Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Models

85% match1 repo22024

Cite this paper

@article{lin2024spoken,
  title  = {Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models},
  author = {Yi-Cheng Lin and Wei-Chih Chen and Hung-yi Lee},
  year   = {2024},
  eprint = {2408.07665},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.07665v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.07665.svg)](https://codeofpaper.com/paper/2408.07665)

Discussion