Position Paper On Diagnostic Uncertainty Estimation from Large Language Models: Next-Word Probability Is Not Pre-test Probability

Yanjun Gao, Skatje Myers, Shan Chen, Dmitriy Dligach, Timothy A Miller +5 more
11/7/2024
cs.AIcs.CL

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

Large language models (LLMs) are being explored for diagnostic decision support, yet their ability to estimate pre-test probabilities, vital for clinical decision-making, remains limited. This study evaluates two LLMs, Mistral-7B and Llama3-70B, using structured electronic health record data on three diagnosis tasks. We examined three current methods of extracting LLM probability estimations and revealed their limitations. We aim to highlight the need for improved techniques in LLM confidence estimation.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

A Survey of Uncertainty Estimation Methods on Large Language Models

88% match2025

A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice

87% match2024

Self-Reported Confidence of Large Language Models in Gastroenterology: Analysis of Commercial, Open-Source, and Quantized Models

86% match2025

Uncertainty Quantification of Large Language Models using Approximate Bayesian Computation

86% match2025

Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review

85% match2025

Systematic Evaluation of Uncertainty Estimation Methods in Large Language Models

85% match2025

Cite this paper

@article{gao2024position,
  title  = {Position Paper On Diagnostic Uncertainty Estimation from Large Language Models: Next-Word Probability Is Not Pre-test Probability},
  author = {Yanjun Gao and Skatje Myers and Shan Chen and Dmitriy Dligach and Timothy A Miller and Danielle Bitterman and Guanhua Chen and Anoop Mayampurath and Matthew Churpek and Majid Afshar},
  year   = {2024},
  eprint = {2411.04962},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.04962v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.04962.svg)](https://codeofpaper.com/paper/2411.04962)

Discussion