ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?

Edisa Lozić, Benjamin Štular
9/14/2023
cs.CLcs.AIcs.CY

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Historical emphasis on writing mastery has shifted with advances in generative AI, especially in scientific writing. This study analysed six AI chatbots for scholarly writing in humanities and archaeology. Using methods that assessed factual correctness and scientific contribution, ChatGPT-4 showed the highest quantitative accuracy, closely followed by ChatGPT-3.5, Bing, and Bard. However, Claude 2 and Aria scored considerably lower. Qualitatively, all AIs exhibited proficiency in merging existing knowledge, but none produced original scientific content. Inter-estingly, our findings suggest ChatGPT-4 might represent a plateau in large language model size. This research emphasizes the unique, intricate nature of human research, suggesting that AI's emulation of human originality in scientific writing is challenging. As of 2023, while AI has transformed content generation, it struggles with original contributions in humanities. This may change as AI chatbots continue to evolve into LLM-powered software.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing

87% match1 repo142023

ChatGPT or academic scientist? Distinguishing authorship with over 99% accuracy using off-the-shelf machine learning tools

85% match2023

Let's have a chat! A Conversation with ChatGPT: Technology, Applications, and Limitations

84% match2023

Chatbot-supported Thesis Writing: An Autoethnographic Report

84% match2023

Do large language models resemble humans in language use?

84% match2023

ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing

83% match2023

Cite this paper

@article{lozi2023chatgpt,
  title  = {ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?},
  author = {Edisa Lozić and Benjamin Štular},
  year   = {2023},
  eprint = {2309.08636},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.08636v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.08636.svg)](https://codeofpaper.com/paper/2309.08636)

Discussion