Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels

Jasper Xian, Saron Samuel, Faraz Khoubsirat, Ronak Pradeep, Md Arafat Sultan +5 more
6/17/2024
cs.IRcs.CLcs.LG

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

We develop a method for training small-scale (under 100M parameter) neural information retrieval models with as few as 10 gold relevance labels. The method depends on generating synthetic queries for documents using a language model (LM), and the key step is that we automatically optimize the LM prompt that is used to generate these queries based on training quality. In experiments with the BIRCO benchmark, we find that models trained with our method outperform RankZephyr and are competitive with RankLLama, both of which are 7B parameter models trained on over 100K labels. These findings point to the power of automatic prompt optimization for synthetic dataset generation.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Augmented Relevance Datasets with Fine-Tuned Small LLMs

84% match2025

Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation

83% match1 repo2022024

Study on LLMs for Promptagator-Style Dense Retriever Training

83% match2025

Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers

83% match1 repo2.1k2022

Crafting Efficient Fine-Tuning Strategies for Large Language Models

83% match2024

Automatic Prompt Selection for Large Language Models

82% match2024

Cite this paper

@article{xian2024prompts,
  title  = {Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels},
  author = {Jasper Xian and Saron Samuel and Faraz Khoubsirat and Ronak Pradeep and Md Arafat Sultan and Radu Florian and Salim Roukos and Avirup Sil and Christopher Potts and Omar Khattab},
  year   = {2024},
  eprint = {2406.11706},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.11706v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.11706.svg)](https://codeofpaper.com/paper/2406.11706)

Discussion