Assessing Good, Bad and Ugly Arguments Generated by ChatGPT: a New Dataset, its Methodology and Associated Tasks

Victor Hugo Nascimento Rocha, Igor Cataneo Silveira, Paulo Pirozelli, Denis Deratani Mauá, Fabio Gagliardi Cozman
6/21/2024
cs.CLcs.AI

No official code found. Best implementation: C4AI/ArGPT (9 stars, last commit 2024-04-08).

Abstract

The recent success of Large Language Models (LLMs) has sparked concerns about their potential to spread misinformation. As a result, there is a pressing need for tools to identify ``fake arguments'' generated by such models. To create these tools, examples of texts generated by LLMs are needed. This paper introduces a methodology to obtain good, bad and ugly arguments from argumentative essays produced by ChatGPT, OpenAI's LLM. We then describe a novel dataset containing a set of diverse arguments, ArGPT. We assess the effectiveness of our dataset and establish baselines for several argumentation-related tasks. Finally, we show that the artificially generated data relates well to human argumentation and thus is useful as a tool to train and test systems for the defined tasks.

View on arXivView PDF

Code Implementations(1)

C4AI/ArGPTCommunity100%

ArGPT: A LLM-based dataset for Argumentation

90Jupyter NotebookJun 15, 20232 years ago

Similar Papers

Assessing Open-Source Large Language Models on Argumentation Mining Subtasks

84% match2024

A comprehensive study of LLM-based argument classification: from LLAMA through GPT-4o to Deepseek-R1

84% match2025

A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets

84% match1 repo152023

ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking about

83% match2023

ArXivBench: When You Should Avoid Using ChatGPT for Academic Writing

83% match1 repo2025

How Persuasive Could LLMs Be? A First Study Combining Linguistic-Rhetorical Analysis and User Experiments

83% match2025

Cite this paper

@article{rocha2024assessing,
  title  = {Assessing Good, Bad and Ugly Arguments Generated by ChatGPT: a New Dataset, its Methodology and Associated Tasks},
  author = {Victor Hugo Nascimento Rocha and Igor Cataneo Silveira and Paulo Pirozelli and Denis Deratani Mauá and Fabio Gagliardi Cozman},
  year   = {2024},
  eprint = {2406.15130},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.15130v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.15130.svg)](https://codeofpaper.com/paper/2406.15130)

Discussion