Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation

Kangwook Jang, Sungnyun Kim, Se-Young Yun, Hoirin Kim
5/19/2023
eess.AScs.CLcs.LG

No official code found. Best implementation: sungnyun/ARMHuBERT (42 stars, Apache-2.0, last commit 2024-08-29).

Abstract

Transformer-based speech self-supervised learning (SSL) models, such as HuBERT, show surprising performance in various speech processing tasks. However, huge number of parameters in speech SSL models necessitate the compression to a more compact model for wider usage in academia or small companies. In this study, we suggest to reuse attention maps across the Transformer layers, so as to remove key and query parameters while retaining the number of layers. Furthermore, we propose a novel masking distillation strategy to improve the student model's speech representation quality. We extend the distillation loss to utilize both masked and unmasked speech frames to fully leverage the teacher model's high-quality representation. Our universal compression strategy yields the student model that achieves phoneme error rate (PER) of 7.72% and word error rate (WER) of 9.96% on the SUPERB benchmark.

View on arXivView PDF

Code Implementations(1)

sungnyun/ARMHuBERTCommunity100%
Apache-2.0

(Interspeech 2023 & ICASSP 2024) Official repository for ARMHuBERT and STaRHuBERT

427Shell, PythonMay 14, 20232 years agoApache-2.0
automatic-speech-recognitiondistillationssl-compression

Similar Papers

SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning

84% match2024

FastWhisper: Adaptive Self-knowledge Distillation for Real-time Automatic Speech Recognition

83% match2026

EAT: Self-Supervised Pre-Training with Efficient Audio Transformer

83% match1 repo2462024

Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models

83% match2023

TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Models

82% match1 repo162025

LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization

82% match2025

Cite this paper

@article{jang2023recycleanddistill,
  title  = {Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation},
  author = {Kangwook Jang and Sungnyun Kim and Se-Young Yun and Hoirin Kim},
  year   = {2023},
  eprint = {2305.11685},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.11685v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.11685.svg)](https://codeofpaper.com/paper/2305.11685)

Discussion