Evaluating LLM-Generated Q&A Test: a Student-Centered Study

Anna Wróblewska, Bartosz Grabek, Jakub Świstak, Daniel Dan
5/10/2025
cs.CLcs.HC

No confident code implementation found for this paper.

Abstract

This research prepares an automatic pipeline for generating reliable question-answer (Q&A) tests using AI chatbots. We automatically generated a GPT-4o-mini-based Q&A test for a Natural Language Processing course and evaluated its psychometric and perceived-quality metrics with students and experts. A mixed-format IRT analysis showed that the generated items exhibit strong discrimination and appropriate difficulty, while student and expert star ratings reflect high overall quality. A uniform DIF check identified two items for review. These findings demonstrate that LLM-generated assessments can match human-authored tests in psychometric performance and user satisfaction, illustrating a scalable approach to AI-assisted assessment development.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

The Future of Learning in the Age of Generative AI: Automated Question Generation and Assessment with Large Language Models

85% match2024

Automated test generation to evaluate tool-augmented LLMs as conversational AI agents

84% match2024

Use Me Wisely: AI-Driven Assessment for LLM Prompting Skills Development

83% match2025

AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs

83% match1 repo12023

The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches

83% match2024

MIRROR: A Novel Approach for the Automated Evaluation of Open-Ended Question Generation

83% match2024

Cite this paper

@article{wrblewska2025evaluating,
  title  = {Evaluating LLM-Generated Q&A Test: a Student-Centered Study},
  author = {Anna Wróblewska and Bartosz Grabek and Jakub Świstak and Daniel Dan},
  year   = {2025},
  eprint = {2505.06591},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.06591v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.06591.svg)](https://codeofpaper.com/paper/2505.06591)

Discussion