Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs

Pranjal Aggarwal, Aman Madaan, Yiming Yang, Mausam
5/19/2023
cs.CL

No official code found. Best implementation: Pranjal2041/AdaptiveConsistency (41 stars, Apache-2.0, last commit 2024-01-30).

Abstract

A popular approach for improving the correctness of output from large language models (LLMs) is Self-Consistency - poll the LLM multiple times and output the most frequent solution. Existing Self-Consistency techniques always generate a constant number of samples per question, where a better approach will be to non-uniformly distribute the available budget based on the amount of agreement in the samples generated so far. In response, we introduce Adaptive-Consistency, a cost-efficient, model-agnostic technique that dynamically adjusts the number of samples per question using a lightweight stopping criterion. Our experiments over 17 reasoning and code generation datasets and three LLMs demonstrate that Adaptive-Consistency reduces sample budget by up to 7.9 times with an average accuracy drop of less than 0.1%. Our code and data are available at https://www.sample-step-by-step.info

View on arXivView PDF

Code Implementations(1)

Pranjal2041/AdaptiveConsistencyCommunity100%
Apache-2.0

Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning with LLMs

416Shell, PythonMay 9, 20232 years agoApache-2.0

Similar Papers

InfiAlign: A Scalable and Sample-Efficient Framework for Aligning LLMs to Enhance Reasoning Capabilities

86% match2025

Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling

85% match1 repo162023

Optimizing Small Language Models for NL2SQL via Chain-of-Thought Fine-Tuning

85% match2026

LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback

85% match2023

CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Models

85% match2023

Aligning CodeLLMs with Direct Preference Optimization

85% match2024

Cite this paper

@article{aggarwal2023lets,
  title  = {Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs},
  author = {Pranjal Aggarwal and Aman Madaan and Yiming Yang and Mausam},
  year   = {2023},
  eprint = {2305.11860},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.11860v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.11860.svg)](https://codeofpaper.com/paper/2305.11860)

Discussion