Can ChatGPT and Bard Generate Aligned Assessment Items? A Reliability Analysis against Human Performance

Abdolvahab Khademi
4/9/2023
cs.CLcs.AI

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

ChatGPT and Bard are AI chatbots based on Large Language Models (LLM) that are slated to promise different applications in diverse areas. In education, these AI technologies have been tested for applications in assessment and teaching. In assessment, AI has long been used in automated essay scoring and automated item generation. One psychometric property that these tools must have to assist or replace humans in assessment is high reliability in terms of agreement between AI scores and human raters. In this paper, we measure the reliability of OpenAI ChatGP and Google Bard LLMs tools against experienced and trained humans in perceiving and rating the complexity of writing prompts. Intraclass correlation (ICC) as a performance metric showed that the inter-reliability of both the OpenAI ChatGPT and the Google Bard were low against the gold standard of human ratings.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Cite this paper

@article{khademi2023chatgpt,
  title  = {Can ChatGPT and Bard Generate Aligned Assessment Items? A Reliability Analysis against Human Performance},
  author = {Abdolvahab Khademi},
  year   = {2023},
  eprint = {2304.05372},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.05372v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.05372.svg)](https://codeofpaper.com/paper/2304.05372)

Discussion