Finetuned Language Models Are Zero-Shot Learners

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu +4 more
9/3/2021
cs.CL

Official code: google-research/FLAN (1,574 stars, Apache-2.0, last commit 2026-04-18).

Abstract

This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning -- finetuning language models on a collection of tasks described via instructions -- substantially improves zero-shot performance on unseen tasks. We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates. We evaluate this instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 tasks that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of finetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.

View on arXivView PDF

Code Implementations(1)

google-research/FLANOfficial100%
Apache-2.0
1,574159Shell, PythonAug 21, 20216 months agoApache-2.0

Similar Papers

Self-Instruct: Aligning Language Models with Self-Generated Instructions

90% match2 repos30.2k2022

Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning

89% match2023

Evaluating the Zero-shot Robustness of Instruction-tuned Language Models

88% match1 repo72023

Instruction Tuned Models are Quick Learners

87% match1 repo52023

Making Pre-trained Language Models Better Few-shot Learners

87% match1 repo7272020

VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization

87% match1 repo92024

Cite this paper

@article{wei2021finetuned,
  title  = {Finetuned Language Models Are Zero-Shot Learners},
  author = {Jason Wei and Maarten Bosma and Vincent Y. Zhao and Kelvin Guu and Adams Wei Yu and Brian Lester and Nan Du and Andrew M. Dai and Quoc V. Le},
  year   = {2021},
  eprint = {2109.01652},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2109.01652}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2109.01652.svg)](https://codeofpaper.com/paper/2109.01652)

Discussion