Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences

Yunjie Ji, Yan Gong, Yiping Peng, Chao Ni, Peiyan Sun +3 more
3/14/2023
cs.CL

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

As a natural language assistant, ChatGPT is capable of performing various tasks, including but not limited to article generation, code completion, and data analysis. Furthermore, ChatGPT has consistently demonstrated a remarkable level of accuracy and reliability in terms of content evaluation, exhibiting the capability of mimicking human preferences. To further explore ChatGPT's potential in this regard, a study is conducted to assess its ability to rank content. In order to do so, a test set consisting of prompts is created, covering a wide range of use cases, and five models are utilized to generate corresponding responses. ChatGPT is then instructed to rank the responses generated by these models. The results on the test set show that ChatGPT's ranking preferences are consistent with human to a certain extent. This preliminary experimental finding implies that ChatGPT's zero-shot ranking capability could be used to reduce annotation pressure in a number of ranking tasks.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents

82% match1 repo6692023

Large Language Models on Wikipedia-Style Survey Generation: an Evaluation in NLP Concepts

82% match2023

Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation

81% match1 repo2023

Benchmarking Cognitive Biases in Large Language Models as Evaluators

81% match1 repo232023

Fusion-Eval: Integrating Assistant Evaluators with LLMs

80% match2023

LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity

80% match2023

Cite this paper

@article{ji2023exploring,
  title  = {Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences},
  author = {Yunjie Ji and Yan Gong and Yiping Peng and Chao Ni and Peiyan Sun and Dongyu Pan and Baochang Ma and Xiangang Li},
  year   = {2023},
  eprint = {2303.07610},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.07610v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.07610.svg)](https://codeofpaper.com/paper/2303.07610)

Discussion