The Effect of Sampling Temperature on Problem Solving in Large Language Models

Matthew Renze, Erhan Guven
2/7/2024
cs.CLcs.AI

Official code: matthewrenze/jhu-llm-temperature (25 stars, BSD-2-Clause, last commit 2024-11-25).

Abstract

In this research study, we empirically investigate the effect of sampling temperature on the performance of Large Language Models (LLMs) on various problem-solving tasks. We created a multiple-choice question-and-answer (MCQA) exam by randomly sampling problems from standard LLM benchmarks. Then, we used nine popular LLMs with five prompt-engineering techniques to solve the MCQA problems while increasing the sampling temperature from 0.0 to 1.6. Despite anecdotal reports to the contrary, our empirical results indicate that changes in temperature from 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks. In addition, these results appear to generalize across LLMs, prompt-engineering techniques, and problem domains. All code, data, and supplemental materials are available on GitHub at: https://github.com/matthewrenze/jhu-llm-temperature

View on arXivView PDF

Code Implementations(1)

matthewrenze/jhu-llm-temperatureOfficial100%
BSD-2-Clause

The Effect of Sampling Temperature on Problem Solving in Large Language Models

252PythonFeb 7, 20241 years agoBSD-2-Clause

Similar Papers

Exploring the Impact of Temperature on Large Language Models:Hot or Cold?

90% match2025

Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs

88% match2025

On the Role of Temperature Sampling in Test-Time Scaling

88% match2025

Optimizing Temperature for Language Models with Multi-Sample Inference

87% match1 repo232025

EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Sampling

84% match1 repo72.8k2024

Thermometer: Towards Universal Calibration for Large Language Models

83% match1 repo132026

Cite this paper

@article{renze2024effect,
  title  = {The Effect of Sampling Temperature on Problem Solving in Large Language Models},
  author = {Matthew Renze and Erhan Guven},
  year   = {2024},
  eprint = {2402.05201},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.05201v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.05201.svg)](https://codeofpaper.com/paper/2402.05201)

Discussion