Examining the Emergence of Deductive Reasoning in Generative Language Models

Peter Belcak, Luca A. Lanzendörfer, Roger Wattenhofer
5/31/2023
cs.CLcs.AIcs.LG

No confident code implementation found for this paper.

Abstract

We conduct a preliminary inquiry into the ability of generative transformer models to deductively reason from premises provided. We observe notable differences in the performance of models coming from different training setups and find that the deductive reasoning ability increases with scale. Further, we discover that the performance generally does not decrease with the length of the deductive chain needed to reach the conclusion, with the exception of OpenAI GPT-3 and GPT-3.5 models. Our study considers a wide variety of transformer-decoder models, ranging from 117 million to 175 billion parameters in size.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Toward Mechanistic Explanation of Deductive Reasoning in Language Models

83% match2025

Transformer-based Language Models for Reasoning in the Description Logic ALCQ

82% match2024

Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning

82% match1 repo12024

Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models

81% match1 repo2025

Implicit Reasoning in Transformers is Reasoning through Shortcuts

81% match1 repo182025

Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts

81% match2024

Cite this paper

@article{belcak2023examining,
  title  = {Examining the Emergence of Deductive Reasoning in Generative Language Models},
  author = {Peter Belcak and Luca A. Lanzendörfer and Roger Wattenhofer},
  year   = {2023},
  eprint = {2306.01009},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.01009v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.01009.svg)](https://codeofpaper.com/paper/2306.01009)

Discussion