Exploring the Potential of Large Language Models to Generate Formative Programming Feedback

Natalie Kiesler, Dominic Lohr, Hieke Keuning
8/31/2023
cs.AIcs.CYcs.SE

No confident code implementation found for this paper. 4 possible matches can be shown below.

Abstract

Ever since the emergence of large language models (LLMs) and related applications, such as ChatGPT, its performance and error analysis for programming tasks have been subject to research. In this work-in-progress paper, we explore the potential of such LLMs for computing educators and learners, as we analyze the feedback it generates to a given input containing program code. In particular, we aim at (1) exploring how an LLM like ChatGPT responds to students seeking help with their introductory programming tasks, and (2) identifying feedback types in its responses. To achieve these goals, we used students' programming sequences from a dataset gathered within a CS1 course as input for ChatGPT along with questions required to elicit feedback and correct solutions. The results show that ChatGPT performs reasonably well for some of the introductory programming tasks and student errors, which means that students can potentially benefit. However, educators should provide guidance on how to use the provided feedback, as it can contain misleading information for novices.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 4 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Evaluating Language Models for Generating and Judging Programming Feedback

92% match2024

Evaluating the Application of Large Language Models to Generate Feedback in Programming Education

91% match2024

Assessing Large Language Models for Automated Feedback Generation in Learning Programming Problem Solving

91% match1 repo12025

Students' Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming

91% match2023

Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models

90% match2023

Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators

90% match2023

Cite this paper

@article{kiesler2023exploring,
  title  = {Exploring the Potential of Large Language Models to Generate Formative Programming Feedback},
  author = {Natalie Kiesler and Dominic Lohr and Hieke Keuning},
  year   = {2023},
  eprint = {2309.00029},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.00029v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.00029.svg)](https://codeofpaper.com/paper/2309.00029)

Discussion