Evaluating GPT's Programming Capability through CodeWars' Katas

Zizhuo Zhang, Lian Wen, Shaoyang Zhang, David Chen, Yanfei Jiang
5/31/2023
cs.AIcs.SE

No confident code implementation found for this paper.

Abstract

In the burgeoning field of artificial intelligence (AI), understanding the capabilities and limitations of programming-oriented models is crucial. This paper presents a novel evaluation of the programming proficiency of Generative Pretrained Transformer (GPT) models, specifically GPT-3.5 and GPT-4, against coding problems of varying difficulty levels drawn from Codewars. The experiments reveal a distinct boundary at the 3kyu level, beyond which these GPT models struggle to provide solutions. These findings led to the proposal of a measure for coding problem complexity that incorporates both problem difficulty and the time required for solution. The research emphasizes the need for validation and creative thinking capabilities in AI models to better emulate human problem-solving techniques. Future work aims to refine this proposed complexity measure, enhance AI models with these suggested capabilities, and develop an objective measure for programming problem difficulty. The results of this research offer invaluable insights for improving AI programming capabilities and advancing the frontier of AI problem-solving abilities.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Evaluating ChatGPT-3.5 Efficiency in Solving Coding Problems of Different Complexity Levels: An Empirical Analysis

83% match1 repo2024

A Preliminary Analysis on the Code Generation Capabilities of GPT-3.5 and Bard AI Models for Java Functions

82% match2023

AI-assisted coding: Experiments with GPT-4

82% match1 repo152023

Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence

82% match2024

ComplexityNet: Increasing LLM Inference Efficiency by Learning Task Complexity

82% match2023

MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams

81% match2025

Cite this paper

@article{zhang2023evaluating,
  title  = {Evaluating GPT's Programming Capability through CodeWars' Katas},
  author = {Zizhuo Zhang and Lian Wen and Shaoyang Zhang and David Chen and Yanfei Jiang},
  year   = {2023},
  eprint = {2306.01784},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.01784v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.01784.svg)](https://codeofpaper.com/paper/2306.01784)

Discussion