Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao
4/20/2019
cs.CL

Official code: namisan/mt-dnn (2,258 stars, MIT, PyTorch, last commit 2024-03-07).

Abstract

This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN) (Liu et al., 2019) for learning text representations across multiple natural language understanding tasks. Although ensemble learning can improve model performance, serving an ensemble of large DNNs such as MT-DNN can be prohibitively expensive. Here we apply the knowledge distillation method (Hinton et al., 2015) in the multi-task learning setting. For each task, we train an ensemble of different MT-DNNs (teacher) that outperforms any single model, and then train a single MT-DNN (student) via multi-task learning to \emph{distill} knowledge from these ensemble teachers. We show that the distilled MT-DNN significantly outperforms the original MT-DNN on 7 out of 9 GLUE tasks, pushing the GLUE benchmark (single model) to 83.7\% (1.5\% absolute improvement\footnote{ Based on the GLUE leaderboard at https://gluebenchmark.com/leaderboard as of April 1, 2019.}). The code and pre-trained models will be made publicly available at https://github.com/namisan/mt-dnn.

View on arXivView PDF

Code Implementations(1)

namisan/mt-dnnOfficial100%
PyTorchMIT

Multi-Task Deep Neural Networks for Natural Language Understanding

2,258407Shell, PythonFeb 19, 20192 years agoMIT
bertdeep-learningmachine-reading-comprehensionmicrosoftmulti-task-learning+5 more

Similar Papers

TinyBERT: Distilling BERT for Natural Language Understanding

89% match1 repo3.2k2019

Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty

87% match1 repo842023

Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning

87% match1 repo142024

Multi-Task Learning with Knowledge Distillation for Dense Prediction

86% match2023

Patient Knowledge Distillation for BERT Model Compression

86% match1 repo2022019

ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation Generalization

86% match1 repo13.0k2023

Cite this paper

@article{liu2019improving,
  title  = {Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding},
  author = {Xiaodong Liu and Pengcheng He and Weizhu Chen and Jianfeng Gao},
  year   = {2019},
  eprint = {1904.09482},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1904.09482}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1904.09482.svg)](https://codeofpaper.com/paper/1904.09482)

Discussion