ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking about

Aman Rangapur, Haoran Wang
4/6/2023
cs.CLcs.IRcs.LG

No confident code implementation found for this paper.

Abstract

Large language models have gained considerable interest for their impressive performance on various tasks. Among these models, ChatGPT developed by OpenAI has become extremely popular among early adopters who even regard it as a disruptive technology in many fields like customer service, education, healthcare, and finance. It is essential to comprehend the opinions of these initial users as it can provide valuable insights into the potential strengths, weaknesses, and success or failure of the technology in different areas. This research examines the responses generated by ChatGPT from different Conversational QA corpora. The study employed BERT similarity scores to compare these responses with correct answers and obtain Natural Language Inference(NLI) labels. Evaluation scores were also computed and compared to determine the overall performance of GPT-3 \& GPT-4. Additionally, the study identified instances where ChatGPT provided incorrect answers to questions, providing insights into areas where the model may be prone to error.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets

88% match1 repo152023

In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT

88% match2023

ChatGPT: Jack of all trades, master of none

87% match1 repo282023

ChatGPT in the Age of Generative AI and Large Language Models: A Concise Survey

87% match1 repo152023

ChatGPT or A Silent Everywhere Helper: A Survey of Large Language Models

86% match2025

The Battle of LLMs: A Comparative Study in Conversational QA Tasks

86% match2024

Cite this paper

@article{rangapur2023chatgptcrawler,
  title  = {ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking about},
  author = {Aman Rangapur and Haoran Wang},
  year   = {2023},
  eprint = {2304.03325},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.03325v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.03325.svg)](https://codeofpaper.com/paper/2304.03325)

Discussion