Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessments

Natalie Kiesler, Daniel Schiffner
8/15/2023
cs.SEcs.AIcs.HC

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

This paper investigates the performance of the Large Language Models (LLMs) ChatGPT-3.5 and GPT-4 in solving introductory programming tasks. Based on the performance, implications for didactic scenarios and assessment formats utilizing LLMs are derived. For the analysis, 72 Python tasks for novice programmers were selected from the free site CodingBat. Full task descriptions were used as input to the LLMs, while the generated replies were evaluated using CodingBat's unit tests. In addition, the general availability of textual explanations and program code was analyzed. The results show high scores of 94.4 to 95.8% correct responses and reliable availability of textual explanations and program code, which opens new ways to incorporate LLMs into programming education and assessment.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors

89% match2023

Feedback-Generation for Programming Exercises With GPT-4

88% match2024

ChatGPT as a Solver and Grader of Programming Exams written in Spanish

88% match2024

Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses

87% match2023

Exploring the Potential of Large Language Models to Generate Formative Programming Feedback

87% match2023

Kattis vs. ChatGPT: Assessment and Evaluation of Programming Tasks in the Age of Artificial Intelligence

87% match2023

Cite this paper

@article{kiesler2023large,
  title  = {Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessments},
  author = {Natalie Kiesler and Daniel Schiffner},
  year   = {2023},
  eprint = {2308.08572},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.08572v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.08572.svg)](https://codeofpaper.com/paper/2308.08572)

Discussion