A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models

Hayeon Lee, Rui Hou, Jongpil Kim, Davis Liang, Sung Ju Hwang +1 more
5/26/2023
cs.CLcs.AI

No official code found. Best implementation: huggingface/transformers (162,097 stars, Apache-2.0, PyTorch, last commit 2026-04-30).

Abstract

Distillation from Weak Teacher (DWT) is a method of transferring knowledge from a smaller, weaker teacher model to a larger student model to improve its performance. Previous studies have shown that DWT can be effective in the vision domain and natural language processing (NLP) pre-training stage. Specifically, DWT shows promise in practical scenarios, such as enhancing new generation or larger models using pre-trained yet older or smaller models and lacking a resource budget. However, the optimal conditions for using DWT have yet to be fully investigated in NLP pre-training. Therefore, this study examines three key factors to optimize DWT, distinct from those used in the vision domain or traditional knowledge distillation. These factors are: (i) the impact of teacher model quality on DWT effectiveness, (ii) guidelines for adjusting the weighting value for DWT loss, and (iii) the impact of parameter remapping as a student model initialization technique for DWT.

View on arXivView PDF

Code Implementations(1)

huggingface/transformersCommunity100%
PyTorchApache-2.0

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

162,09733,718Shell, PythonOct 29, 20185 months agoApache-2.0
audiodeep-learningdeepseekgemmaglm+14 more

Similar Papers

EvoText: Enhancing Natural Language Generation Models via Self-Escalation Learning for Up-to-Date Knowledge and Improved Performance

88% match2023

TinyBERT: Distilling BERT for Natural Language Understanding

86% match1 repo3.2k2019

Patient Knowledge Distillation for BERT Model Compression

86% match1 repo2022019

CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval

85% match2023

Knowledge Editing for Large Language Models: A Survey

85% match1 repo1.2k2023

Unsupervised Domain Adaption Harnessing Vision-Language Pre-training

85% match1 repo382024

Cite this paper

@article{lee2023study,
  title  = {A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models},
  author = {Hayeon Lee and Rui Hou and Jongpil Kim and Davis Liang and Sung Ju Hwang and Alexander Min},
  year   = {2023},
  eprint = {2305.18239},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.18239v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.18239.svg)](https://codeofpaper.com/paper/2305.18239)

Discussion