Gender Inclusivity Fairness Index (GIFI): A Multilevel Framework for Evaluating Gender Diversity in Large Language Models

Zhengyang Shan, Emily Ruth Diana, Jiawei Zhou
6/18/2025
cs.CL

No official code found. Best implementation: ZhengyangShan/GIFI (3 stars, MIT, last commit 2025-07-15).

Abstract

We present a comprehensive evaluation of gender fairness in large language models (LLMs), focusing on their ability to handle both binary and non-binary genders. While previous studies primarily focus on binary gender distinctions, we introduce the Gender Inclusivity Fairness Index (GIFI), a novel and comprehensive metric that quantifies the diverse gender inclusivity of LLMs. GIFI consists of a wide range of evaluations at different levels, from simply probing the model with respect to provided gender pronouns to testing various aspects of model generation and cognitive behaviors under different gender assumptions, revealing biases associated with varying gender identifiers. We conduct extensive evaluations with GIFI on 22 prominent open-source and proprietary LLMs of varying sizes and capabilities, discovering significant variations in LLMs' gender inclusivity. Our study highlights the importance of improving LLMs' inclusivity, providing a critical benchmark for future advancements in gender fairness in generative models.

View on arXivView PDF

Code Implementations(1)

ZhengyangShan/GIFICommunity76%
MIT

This includes the implementation of Gender Inclusivity Fairness Index (GIFI): A Multilevel Framework for Evaluating Gender Diversity in Large Language Models.

31Shell, PythonNov 9, 20241 years agoMIT

Similar Papers

GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models

90% match1 repo272024

Generalizing Fairness to Generative Language Models via Reformulation of Non-discrimination Criteria

88% match1 repo12024

A database to support the evaluation of gender biases in GPT-4o output

86% match2025

From 'Showgirls' to 'Performers': Fine-tuning with Gender-inclusive Language for Bias Reduction in LLMs

86% match2024

Gender Bias in Decision-Making with Large Language Models: A Study of Relationship Conflicts

86% match1 repo22024

Gender Bias in Large Language Models across Multiple Languages

86% match2024

Cite this paper

@article{shan2025gender,
  title  = {Gender Inclusivity Fairness Index (GIFI): A Multilevel Framework for Evaluating Gender Diversity in Large Language Models},
  author = {Zhengyang Shan and Emily Ruth Diana and Jiawei Zhou},
  year   = {2025},
  eprint = {2506.15568},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.15568v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.15568.svg)](https://codeofpaper.com/paper/2506.15568)

Discussion