AI-assisted coding: Experiments with GPT-4

Russell A Poldrack, Thomas Lu, Gašper Beguš
4/25/2023
cs.AIcs.SE

No official code found. Best implementation: poldrack/ai-coding-experiments (15 stars, last commit 2025-07-05).

Abstract

Artificial intelligence (AI) tools based on large language models have acheived human-level performance on some computer programming tasks. We report several experiments using GPT-4 to generate computer code. These experiments demonstrate that AI code generation using the current generation of tools, while powerful, requires substantial human validation to ensure accurate performance. We also demonstrate that GPT-4 refactoring of existing code can significantly improve that code along several established metrics for code quality, and we show that GPT-4 can generate tests with substantial coverage, but that many of the tests fail when applied to the associated code. These findings suggest that while AI coding tools are very powerful, they still require humans in the loop to ensure validity and accuracy of the results.

View on arXivView PDF

Code Implementations(1)

poldrack/ai-coding-experimentsCommunity100%

Code and data for AI-assisted coding experiments

152Lua, TeX1 years ago

Similar Papers

Comparing large language models and human programmers for generating programming code

89% match2024

Assessing the Promise and Pitfalls of ChatGPT for Automated Code Generation

88% match1 repo52023

Testing LLMs on Code Generation with Varying Levels of Prompt Specificity

88% match2023

Towards Coding Social Science Datasets with Language Models

87% match2023

Disrupting Test Development with AI Assistants

86% match2024

AIGCodeSet: A New Annotated Dataset for AI Generated Code Detection

86% match2024

Cite this paper

@article{poldrack2023aiassisted,
  title  = {AI-assisted coding: Experiments with GPT-4},
  author = {Russell A Poldrack and Thomas Lu and Gašper Beguš},
  year   = {2023},
  eprint = {2304.13187},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.13187v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.13187.svg)](https://codeofpaper.com/paper/2304.13187)

Discussion