BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

Asa Cooper Stickland, Iain Murray
2/7/2019
cs.LGcs.CLstat.ML

Official code: AsaCooperStickland/Bert-n-Pals (86 stars, MIT, last commit 2019-05-09).

Abstract

Multi-task learning shares information between related tasks, sometimes reducing the number of parameters required. State-of-the-art results across multiple natural language understanding tasks in the GLUE benchmark have previously used transfer from a single large task: unsupervised pre-training with BERT, where a separate BERT model was fine-tuned for each task. We explore multi-task approaches that share a single BERT model with a small number of additional task-specific parameters. Using new adaptation modules, PALs or `projected attention layers', we match the performance of separately fine-tuned models on the GLUE benchmark with roughly 7 times fewer parameters, and obtain state-of-the-art results on the Recognizing Textual Entailment dataset.

View on arXivView PDF

Code Implementations(1)

AsaCooperStickland/Bert-n-PalsOfficial100%
MIT

Pytorch implementation of Bert and Pals: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning (https://arxiv.org/abs/1902.02671)

8625PythonApr 20, 20197 years agoMIT

Similar Papers

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

89% match2 repos6.6k2021

SkillNet-X: A Multilingual Multitask Model with Sparsely Activated Skills

88% match2023

TaskWeb: Selecting Better Source Tasks for Multi-task NLP

88% match1 repo52023

Hierarchical Prompt Learning for Multi-Task Learning

88% match2023

Multi-Task Deep Neural Networks for Natural Language Understanding

88% match1 repo2.3k2019

UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions

87% match2023

Cite this paper

@article{stickland2019bert,
  title  = {BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning},
  author = {Asa Cooper Stickland and Iain Murray},
  year   = {2019},
  eprint = {1902.02671},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1902.02671}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1902.02671.svg)](https://codeofpaper.com/paper/1902.02671)

Discussion