Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking

Shengyao Zhuang, Bing Liu, Bevan Koopman, Guido Zuccon
10/20/2023
cs.IRcs.CL

Official code: ielab/llm-qlm (17 stars, Apache-2.0, last commit 2023-10-26).

Abstract

In the field of information retrieval, Query Likelihood Models (QLMs) rank documents based on the probability of generating the query given the content of a document. Recently, advanced large language models (LLMs) have emerged as effective QLMs, showcasing promising ranking capabilities. This paper focuses on investigating the genuine zero-shot ranking effectiveness of recent LLMs, which are solely pre-trained on unstructured text data without supervised instruction fine-tuning. Our findings reveal the robust zero-shot ranking ability of such LLMs, highlighting that additional instruction fine-tuning may hinder effectiveness unless a question generation task is present in the fine-tuning dataset. Furthermore, we introduce a novel state-of-the-art ranking system that integrates LLM-based QLMs with a hybrid zero-shot retriever, demonstrating exceptional effectiveness in both zero-shot and few-shot scenarios. We make our codebase publicly available at https://github.com/ielab/llm-qlm.

View on arXivView PDF

Code Implementations(1)

ielab/llm-qlmOfficial100%
Apache-2.0

Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking

170PythonOct 18, 20232 years agoApache-2.0

Similar Papers

GAR-meets-RAG Paradigm for Zero-Shot Information Retrieval

88% match2023

A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models

87% match1 repo2102023

Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model

85% match2024

RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models

85% match1 repo6602023

Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents

85% match1 repo6692023

Zero-shot Audio Topic Reranking using Large Language Models

85% match2023

Cite this paper

@article{zhuang2023opensource,
  title  = {Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking},
  author = {Shengyao Zhuang and Bing Liu and Bevan Koopman and Guido Zuccon},
  year   = {2023},
  eprint = {2310.13243},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.13243v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.13243.svg)](https://codeofpaper.com/paper/2310.13243)

Discussion