The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance

Kyle Moore, Jesse Roberts, Thao Pham, Oseremhen Ewaleifoh, Doug Fisher
6/17/2024
cs.CLcs.AI

No confident code implementation found for this paper.

Abstract

Cloze testing is a common method for measuring the behavior of large language models on a number of benchmark tasks. Using the MMLU dataset, we show that the base-rate probability (BRP) differences across answer tokens are significant and affect task performance ie. guess A if uncertain. We find that counterfactual prompting does sufficiently mitigate the BRP effect. The BRP effect is found to have a similar effect to test taking strategies employed by humans leading to the conflation of task performance and test-taking ability. We propose the Nvr-X-MMLU task, a variation of MMLU, which helps to disambiguate test-taking ability from task performance and reports the latter.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks

85% match1 repo42025

Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks

84% match2024

Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory

83% match1 repo182025

Forget What You Know about LLMs Evaluations -- LLMs are Like a Chameleon

83% match1 repo32025

Changing Answer Order Can Decrease MMLU Accuracy

83% match2024

Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdouts

83% match2024

Cite this paper

@article{moore2024baserate,
  title  = {The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance},
  author = {Kyle Moore and Jesse Roberts and Thao Pham and Oseremhen Ewaleifoh and Doug Fisher},
  year   = {2024},
  eprint = {2406.11634},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.11634v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.11634.svg)](https://codeofpaper.com/paper/2406.11634)

Discussion