Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation

Minglun Han, Feilong Chen, Jing Shi, Shuang Xu, Bo Xu
1/30/2023
cs.CLcs.AIcs.SD

No official code found. Best implementation: MingLunHan/CIF-PyTorch (78 stars, Apache-2.0, PyTorch, last commit 2025-01-09).

Abstract

Large-scale pre-trained language models (PLMs) have shown great potential in natural language processing tasks. Leveraging the capabilities of PLMs to enhance automatic speech recognition (ASR) systems has also emerged as a promising research direction. However, previous works may be limited by the inflexible structures of PLMs and the insufficient utilization of PLMs. To alleviate these problems, we propose the hierarchical knowledge distillation (HKD) on the continuous integrate-and-fire (CIF) based ASR models. To transfer knowledge from PLMs to the ASR models, HKD employs cross-modal knowledge distillation with contrastive loss at the acoustic level and knowledge distillation with regression loss at the linguistic level. Compared with the original CIF-based model, our method achieves 15% and 9% relative error rate reduction on the AISHELL-1 and LibriSpeech datasets, respectively.

View on arXivView PDF

Code Implementations(1)

MingLunHan/CIF-PyTorchCommunity100%
PyTorchApache-2.0

[ICASSP 2020] CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition (A PyTorch implementation of Continuous Integrate-and-Fire mechanism).

786PythonOct 27, 20211 years agoApache-2.0
alignmentasrautomatic-speech-recognitioncifcontinuous-integrate-and-fire+6 more

Similar Papers

Adaptive Knowledge Distillation between Text and Speech Pre-trained Models

86% match2023

TinyBERT: Distilling BERT for Natural Language Understanding

86% match1 repo3.2k2019

Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method

85% match1 repo342023

CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training

84% match2023

DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognition

83% match1 repo92023

Cross-Modal Knowledge Distillation for Speech Large Language Models

83% match2025

Cite this paper

@article{han2023knowledge,
  title  = {Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation},
  author = {Minglun Han and Feilong Chen and Jing Shi and Shuang Xu and Bo Xu},
  year   = {2023},
  eprint = {2301.13003},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2301.13003v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2301.13003.svg)](https://codeofpaper.com/paper/2301.13003)

Discussion