Multi-Task Deep Neural Networks for Natural Language Understanding

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao
1/31/2019
cs.CL

Official code: namisan/mt-dnn (2,258 stars, MIT, PyTorch, last commit 2024-03-07).

Abstract

In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization effect that leads to more general representations in order to adapt to new tasks and domains. MT-DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirectional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI, SciTail, and eight out of nine GLUE tasks, pushing the GLUE benchmark to 82.7% (2.2% absolute improvement). We also demonstrate using the SNLI and SciTail datasets that the representations learned by MT-DNN allow domain adaptation with substantially fewer in-domain labels than the pre-trained BERT representations. The code and pre-trained models are publicly available at https://github.com/namisan/mt-dnn.

View on arXivView PDF

Code Implementations(1)

namisan/mt-dnnOfficial100%
PyTorchMIT

Multi-Task Deep Neural Networks for Natural Language Understanding

2,258407Shell, PythonFeb 19, 20192 years agoMIT
bertdeep-learningmachine-reading-comprehensionmicrosoftmulti-task-learning+5 more

Similar Papers

BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

88% match1 repo862019

A Pragmatics-Centered Evaluation Framework for Natural Language Understanding

86% match1 repo212019

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

84% match1 repo40.0k2018

TransPrompt v2: A Transferable Prompting Framework for Cross-task Text Classification

84% match2023

UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions

84% match2023

The Natural Language Decathlon: Multitask Learning as Question Answering

84% match1 repo2.3k2018

Cite this paper

@article{liu2019multitask,
  title  = {Multi-Task Deep Neural Networks for Natural Language Understanding},
  author = {Xiaodong Liu and Pengcheng He and Weizhu Chen and Jianfeng Gao},
  year   = {2019},
  eprint = {1901.11504},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1901.11504}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1901.11504.svg)](https://codeofpaper.com/paper/1901.11504)

Discussion