Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithms

Mayank Soni, Vincent Wade
3/30/2023
cs.CL

No confident code implementation found for this paper.

Abstract

Large Language Models (LLMs) have gathered significant attention due to their impressive performance on a variety of tasks. ChatGPT, developed by OpenAI, is a recent addition to the family of language models and is being called a disruptive technology by a few, owing to its human-like text-generation capabilities. Although, many anecdotal examples across the internet have evaluated ChatGPT's strength and weakness, only a few systematic research studies exist. To contribute to the body of literature of systematic research on ChatGPT, we evaluate the performance of ChatGPT on Abstractive Summarization by the means of automated metrics and blinded human reviewers. We also build automatic text classifiers to detect ChatGPT generated summaries. We found that while text classification algorithms can distinguish between real and generated summaries, humans are unable to distinguish between real summaries and those produced by ChatGPT.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

A Critical Review of Large Language Models: Sensitivity, Bias, and the Path Toward Specialized AI

88% match2023

Findings of the Counter Turing Test: AI-Generated Text Detection

87% match2026

The Imitation Game: Detecting Human and AI-Generated Texts in the Era of ChatGPT and BARD

87% match1 repo112023

GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content

87% match1 repo92023

The Science of Detecting LLM-Generated Texts

87% match2023

Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?

86% match1 repo72023

Cite this paper

@article{soni2023comparing,
  title  = {Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithms},
  author = {Mayank Soni and Vincent Wade},
  year   = {2023},
  eprint = {2303.17650},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.17650v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.17650.svg)](https://codeofpaper.com/paper/2303.17650)

Discussion