Characterizing Selective Refusal Bias in Large Language Models

Adel Khorramrouz, Sharon Levy
10/31/2025
cs.CLcs.CY

No official code found. Best implementation: Akhorramrouz/Selective-Refusal-Bias (0 stars, MIT, last commit 2025-10-30).

Abstract

Safety guardrails in large language models(LLMs) are developed to prevent malicious users from generating toxic content at a large scale. However, these measures can inadvertently introduce or reflect new biases, as LLMs may refuse to generate harmful content targeting some demographic groups and not others. We explore this selective refusal bias in LLM guardrails through the lens of refusal rates of targeted individual and intersectional demographic groups, types of LLM responses, and length of generated refusals. Our results show evidence of selective refusal bias across gender, sexual orientation, nationality, and religion attributes. This leads us to investigate additional safety implications via an indirect attack, where we target previously refused groups. Our findings emphasize the need for more equitable and robust performance in safety guardrails across demographic groups.

View on arXivView PDF

Code Implementations(1)

Akhorramrouz/Selective-Refusal-BiasCommunity73%
MIT

Characterizing Selective Refusal Bias in Large Language Models

00Oct 30, 202511 months agoMIT

Similar Papers

Identifying Features Associated with Bias Against 93 Stigmatized Groups in Language Models and Guardrail Model Safety Mitigation

88% match1 repo12025

Are Large Language Models Really Bias-Free? Jailbreak Prompts for Assessing Adversarial Robustness to Bias Elicitation

87% match1 repo2024

Analyzing Bias in False Refusal Behavior of Large Language Models for Hate Speech Detoxification

86% match2026

Investigating Bias Representations in Llama 2 Chat via Activation Steering

85% match2024

Refusal Behavior in Large Language Models: A Nonlinear Perspective

85% match1 repo32025

Measuring Stereotype and Deviation Biases in Large Language Models

85% match2025

Cite this paper

@article{khorramrouz2025characterizing,
  title  = {Characterizing Selective Refusal Bias in Large Language Models},
  author = {Adel Khorramrouz and Sharon Levy},
  year   = {2025},
  eprint = {2510.27087},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.27087v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.27087.svg)](https://codeofpaper.com/paper/2510.27087)

Discussion