A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, Richard Socher
11/5/2016
cs.CLcs.AI

Official code: hassyGo/charNgram2vec (23 stars, MIT, last commit 2023-11-01).

Abstract

Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. Higher layers include shortcut connections to lower-level task predictions to reflect linguistic hierarchies. We use a simple regularization term to allow for optimizing all model weights to improve one task's loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end model obtains state-of-the-art or competitive results on five different tasks from tagging, parsing, relatedness, and entailment tasks.

View on arXivView PDF

Code Implementations(1)

hassyGo/charNgram2vecOfficial100%
MIT

Pre-training character n-gram embeddings

234C++, MakefileDec 8, 20162 years agoMIT
embeddings

Similar Papers

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

87% match2 repos6.6k2021

FonMTL: Towards Multitask Learning for the Fon Language

87% match1 repo52023

BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

87% match1 repo862019

Hierarchical Prompt Learning for Multi-Task Learning

86% match2023

SkillNet-X: A Multilingual Multitask Model with Sparsely Activated Skills

86% match2023

Multi-task Active Learning for Pre-trained Transformer-based Models

85% match1 repo42022

Cite this paper

@article{hashimoto2016joint,
  title  = {A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks},
  author = {Kazuma Hashimoto and Caiming Xiong and Yoshimasa Tsuruoka and Richard Socher},
  year   = {2016},
  eprint = {1611.01587},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1611.01587}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1611.01587.svg)](https://codeofpaper.com/paper/1611.01587)

Discussion