Comparing large language models and human programmers for generating programming code

Wenpin Hou, Zhicheng Ji
3/1/2024
cs.SEcs.AIcs.CL

No confident code implementation found for this paper. 4 possible matches can be shown below.

Abstract

We systematically evaluated the performance of seven large language models in generating programming code using various prompt strategies, programming languages, and task difficulties. GPT-4 substantially outperforms other large language models, including Gemini Ultra and Claude 2. The coding performance of GPT-4 varies considerably with different prompt strategies. In most LeetCode and GeeksforGeeks coding contests evaluated in this study, GPT-4 employing the optimal prompt strategy outperforms 85 percent of human participants. Additionally, GPT-4 demonstrates strong capabilities in translating code between different programming languages and in learning from past errors. The computational efficiency of the code generated by GPT-4 is comparable to that of human programmers. These results suggest that GPT-4 has the potential to serve as a reliable assistant in programming code generation and software development.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 4 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Evaluating the Application of Large Language Models to Generate Feedback in Programming Education

86% match2024

Holistic Evaluation of State-of-the-Art LLMs for Code Generation

86% match2025

Examination of Code generated by Large Language Models

86% match2024

An evaluation of LLM code generation capabilities through graded exercises

86% match2024

The Program Testing Ability of Large Language Models for Code

86% match2023

Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks

85% match2024

Cite this paper

@article{hou2024comparing,
  title  = {Comparing large language models and human programmers for generating programming code},
  author = {Wenpin Hou and Zhicheng Ji},
  year   = {2024},
  eprint = {2403.00894},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2403.00894v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.00894.svg)](https://codeofpaper.com/paper/2403.00894)

Discussion