Self-Reported Confidence of Large Language Models in Gastroenterology: Analysis of Commercial, Open-Source, and Quantized Models

Nariman Naderi, Seyed Amir Ahmad Safavi-Naini, Thomas Savage, Zahra Atf, Peter Lewis +2 more
3/24/2025
cs.CLcs.AIcs.HC

No confident code implementation found for this paper. 4 possible matches can be shown below.

Abstract

This study evaluated self-reported response certainty across several large language models (GPT, Claude, Llama, Phi, Mistral, Gemini, Gemma, and Qwen) using 300 gastroenterology board-style questions. The highest-performing models (GPT-o1 preview, GPT-4o, and Claude-3.5-Sonnet) achieved Brier scores of 0.15-0.2 and AUROC of 0.6. Although newer models demonstrated improved performance, all exhibited a consistent tendency towards overconfidence. Uncertainty estimation presents a significant challenge to the safe use of LLMs in healthcare. Keywords: Large Language Models; Confidence Elicitation; Artificial Intelligence; Gastroenterology; Uncertainty Quantification

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 4 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Position Paper On Diagnostic Uncertainty Estimation from Large Language Models: Next-Word Probability Is Not Pre-test Probability

86% match2024

Evaluating Prompt Engineering Techniques for Accuracy and Confidence Elicitation in Medical LLMs

86% match1 repo2025

Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey

85% match2025

Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation

85% match2026

Uncertainty Estimation of Large Language Models in Medical Question Answering

84% match2024

Trustworthy Agents for Electronic Health Records through Confidence Estimation

84% match2025

Cite this paper

@article{naderi2025selfreported,
  title  = {Self-Reported Confidence of Large Language Models in Gastroenterology: Analysis of Commercial, Open-Source, and Quantized Models},
  author = {Nariman Naderi and Seyed Amir Ahmad Safavi-Naini and Thomas Savage and Zahra Atf and Peter Lewis and Girish Nadkarni and Ali Soroush},
  year   = {2025},
  eprint = {2503.18562},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2503.18562v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2503.18562.svg)](https://codeofpaper.com/paper/2503.18562)

Discussion