GPT-4 Can't Reason

Konstantine Arkoudas
7/21/2023
cs.CL

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

GPT-4 was released in March 2023 to wide acclaim, marking a very substantial improvement across the board over GPT-3.5 (OpenAI's previously best model, which had powered the initial release of ChatGPT). However, despite the genuinely impressive improvement, there are good reasons to be highly skeptical of GPT-4's ability to reason. This position paper discusses the nature of reasoning; criticizes the current formulation of reasoning problems in the NLP community, as well as the way in which LLM reasoning performance is currently evaluated; introduces a small collection of 21 diverse reasoning problems; and performs a detailed qualitative evaluation of GPT-4's performance on those problems. Based on this analysis, the paper concludes that, despite its occasional flashes of analytical brilliance, GPT-4 at present is utterly incapable of reasoning.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

86% match1 repo312023

GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts

85% match2023

Competition-Level Problems are Effective LLM Evaluators

81% match2023

Quantifying the Capability Boundary of DeepSeek Models: An Application-Driven Performance Analysis

80% match2025

Hermes 4 Technical Report

80% match2025

Gpt-4: A Review on Advancements and Opportunities in Natural Language Processing

79% match2023

Cite this paper

@article{arkoudas2023gpt,
  title  = {GPT-4 Can't Reason},
  author = {Konstantine Arkoudas},
  year   = {2023},
  eprint = {2308.03762},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.03762v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.03762.svg)](https://codeofpaper.com/paper/2308.03762)

Discussion