Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

Lorenz Kuhn, Yarin Gal, Sebastian Farquhar
2/19/2023
cs.CLcs.AIcs.LG

No official code found. Best implementation: lorenzkuhn/semantic_uncertainty (186 stars, MIT, last commit 2024-06-20).

Abstract

We introduce a method to measure uncertainty in large language models. For tasks like question answering, it is essential to know when we can trust the natural language outputs of foundation models. We show that measuring uncertainty in natural language is challenging because of "semantic equivalence" -- different sentences can mean the same thing. To overcome these challenges we introduce semantic entropy -- an entropy which incorporates linguistic invariances created by shared meanings. Our method is unsupervised, uses only a single model, and requires no modifications to off-the-shelf language models. In comprehensive ablation studies we show that the semantic entropy is more predictive of model accuracy on question answering data sets than comparable baselines.

View on arXivView PDF

Code Implementations(1)

lorenzkuhn/semantic_uncertaintyCommunity100%
MIT
18624Shell, PythonFeb 28, 20232 years agoMIT

Similar Papers

A statistically consistent measure of semantic uncertainty using Language Models

93% match2025

Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities

89% match1 repo372026

Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models

89% match1 repo132025

Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity

89% match1 repo112025

Uncertainty in Natural Language Generation: From Theory to Applications

88% match2023

ESI: Epistemic Uncertainty Quantification via Semantic-preserving Intervention for Large Language Models

88% match1 repo2025

Cite this paper

@article{kuhn2023semantic,
  title  = {Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation},
  author = {Lorenz Kuhn and Yarin Gal and Sebastian Farquhar},
  year   = {2023},
  eprint = {2302.09664},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.09664v3},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.09664.svg)](https://codeofpaper.com/paper/2302.09664)

Discussion