Instruction-Following Evaluation for Large Language Models

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu +3 more
11/14/2023
cs.CLcs.AIcs.LG

Abstract

One core capability of Large Language Models (LLMs) is to follow natural language instructions. However, the evaluation of such abilities is not standardized: Human evaluations are expensive, slow, and not objectively reproducible, while LLM-based auto-evaluation is potentially biased or limited by the ability of the evaluator LLM. To overcome these issues, we introduce Instruction-Following Eval (IFEval) for large language models. IFEval is a straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword of AI at least 3 times". We identified 25 types of those verifiable instructions and constructed around 500 prompts, with each prompt containing one or more verifiable instructions. We show evaluation results of two widely available LLMs on the market. Our code and data can be found at https://github.com/google-research/google-research/tree/master/instruction_following_eval

View on arXivView PDF

Code Implementations(2)

google-research/google-researchOfficial100%
Apache-2.0

Google Research

37,8258,396C, ROct 4, 20184 months agoApache-2.0
aimachine-learningresearch
princeton-nlp/LLMBarCommunity63%
MIT

[ICLR 2024] Evaluating Large Language Models at Evaluating Instruction Following

13811PythonOct 12, 20232 years agoMIT
evaluationllmnlp

Cite this paper

@article{zhou2023instructionfollowing,
  title  = {Instruction-Following Evaluation for Large Language Models},
  author = {Jeffrey Zhou and Tianjian Lu and Swaroop Mishra and Siddhartha Brahma and Sujoy Basu and Yi Luan and Denny Zhou and Le Hou},
  year   = {2023},
  eprint = {2311.07911},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.07911v1}
}

Discussion