Leveraging In-Context Learning for Political Bias Testing of LLMs

Patrick Haller, Jannis Vamvas, Rico Sennrich, Lena A. Jäger
6/27/2025
cs.CL

No official code found. Best implementation: ZurichNLP/questionnaire (1 star, MIT, last commit 2025-05-30).

Abstract

A growing body of work has been querying LLMs with political questions to evaluate their potential biases. However, this probing method has limited stability, making comparisons between models unreliable. In this paper, we argue that LLMs need more context. We propose a new probing task, Questionnaire Modeling (QM), that uses human survey data as in-context examples. We show that QM improves the stability of question-based bias evaluation, and demonstrate that it may be used to compare instruction-tuned models to their base versions. Experiments with LLMs of various sizes indicate that instruction tuning can indeed change the direction of bias. Furthermore, we observe a trend that larger models are able to leverage in-context examples more effectively, and generally exhibit smaller bias scores in QM. Data and code are publicly available.

View on arXivView PDF

Code Implementations(1)

ZurichNLP/questionnaireCommunity59%
MIT

Code and data for the paper "Leveraging In-Context Learning for Political Bias Testing of LLMs"

10R, PythonJun 13, 20241 years agoMIT

Similar Papers

Measuring Political Bias in Large Language Models: What Is Said and How It Is Said

85% match2024

Political Plasticity: An Analysis of Ideological Adaptability in Large Language Models

83% match2026

Polar: A Benchmark for Evaluating Political Bias in LLMs

83% match2026

Steering Towards Fairness: Mitigating Political Bias in LLMs

83% match2025

Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models

83% match1 repo12025

United in Diversity? Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Elections

82% match2024

Cite this paper

@article{haller2025leveraging,
  title  = {Leveraging In-Context Learning for Political Bias Testing of LLMs},
  author = {Patrick Haller and Jannis Vamvas and Rico Sennrich and Lena A. Jäger},
  year   = {2025},
  eprint = {2506.22232},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.22232v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.22232.svg)](https://codeofpaper.com/paper/2506.22232)

Discussion