GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model

Shicheng Tan, Weng Lam Tam, Yuanchun Wang, Wenwen Gong, Yang Yang +7 more
6/11/2023
cs.CLcs.AI

No official code found. Best implementation: aitsc/GLMKD (34 stars, MIT, last commit 2023-08-04).

Abstract

Currently, the reduction in the parameter scale of large-scale pre-trained language models (PLMs) through knowledge distillation has greatly facilitated their widespread deployment on various devices. However, the deployment of knowledge distillation systems faces great challenges in real-world industrial-strength applications, which require the use of complex distillation methods on even larger-scale PLMs (over 10B), limited by memory on GPUs and the switching of methods. To overcome these challenges, we propose GKD, a general knowledge distillation framework that supports distillation on larger-scale PLMs using various distillation methods. With GKD, developers can build larger distillation models on memory-limited GPUs and easily switch and combine different distillation methods within a single framework. Experimental results show that GKD can support the distillation of at least 100B-scale PLMs and 25 mainstream methods on 8 NVIDIA A100 (40GB) GPUs.

View on arXivView PDF

Code Implementations(1)

aitsc/GLMKDCommunity100%
MIT

Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method ; GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model

341Shell, PythonMay 12, 20233 years agoMIT

Similar Papers

Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method

89% match1 repo342023

DDK: Distilling Domain Knowledge for Efficient Large Language Models

89% match2026

Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application

88% match2024

Can a student Large Language Model perform as well as it's teacher?

88% match2023

Pre-training Distillation for Large Language Models: A Design Space Exploration

87% match2024

Benchmarking Distilled Language Models: Performance and Efficiency in Resource-Constrained Settings

87% match2026

Cite this paper

@article{tan2023gkd,
  title  = {GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model},
  author = {Shicheng Tan and Weng Lam Tam and Yuanchun Wang and Wenwen Gong and Yang Yang and Hongyin Tang and Keqing He and Jiahao Liu and Jingang Wang and Shu Zhao and Peng Zhang and Jie Tang},
  year   = {2023},
  eprint = {2306.06629},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.06629v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.06629.svg)](https://codeofpaper.com/paper/2306.06629)

Discussion