Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers

Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie +3 more
5/21/2023
cs.CL

Official code: gonglinyuan/metro_t0 (22 stars, last commit 2023-11-01).

Abstract

This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5. We study various designs to pretrain T5 using an auxiliary model to construct more challenging token replacements for the main model to denoise. Key aspects under study include the decoding target, the location of the RTD head, and the masking pattern. Based on these studies, we develop a new model, METRO-T0, which is pretrained using the redesigned ELECTRA-Style pretraining strategies and then prompt-finetuned on a mixture of NLP tasks. METRO-T0 outperforms all similar-sized baselines on prompted NLP benchmarks, such as T0 Eval and MMLU, and rivals the state-of-the-art T0-11B model with only 8% of its parameters. Our analysis on model's neural activation and parameter sensitivity reveals that the effectiveness of METRO-T0 stems from more balanced contribution of parameters and better utilization of their capacity. The code and model checkpoints are available at https://github.com/gonglinyuan/metro_t0.

View on arXivView PDF

Code Implementations(1)

gonglinyuan/metro_t0Official100%

Code repo for "Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers" (ACL 2023)

223Shell, CythonMay 6, 20232 years ago

Similar Papers

Octopus: A Multitask Model and Toolkit for Arabic Natural Language Generation

84% match2023

ELECTRA is a Zero-Shot Learner, Too

84% match1 repo92022

EvoText: Enhancing Natural Language Generation Models via Self-Escalation Learning for Up-to-Date Knowledge and Improved Performance

84% match2023

Gradient Ascent Post-training Enhances Language Model Generalization

83% match1 repo292023

Long-Text-to-Image Generation via Compositional Prompt Decomposition

83% match2026

WC-SBERT: Zero-Shot Text Classification via SBERT with Self-Training for Wikipedia Categories

83% match1 repo32023

Cite this paper

@article{gong2023modelgenerated,
  title  = {Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers},
  author = {Linyuan Gong and Chenyan Xiong and Xiaodong Liu and Payal Bajaj and Yiqing Xie and Alvin Cheung and Jianfeng Gao and Xia Song},
  year   = {2023},
  eprint = {2305.12567},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.12567v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.12567.svg)](https://codeofpaper.com/paper/2305.12567)

Discussion