Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning

Guanrou Yang, Ziyang Ma, Zhisheng Zheng, Yakun Song, Zhikang Niu +1 more
9/25/2023
cs.CLcs.AIcs.LG

No official code found. Best implementation: yanghaha0908/FastHuBERT (100 stars, MIT, last commit 2024-11-20).

Abstract

Recent years have witnessed significant advancements in self-supervised learning (SSL) methods for speech-processing tasks. Various speech-based SSL models have been developed and present promising performance on a range of downstream tasks including speech recognition. However, existing speech-based SSL models face a common dilemma in terms of computational cost, which might hinder their potential application and in-depth academic research. To address this issue, we first analyze the computational cost of different modules during HuBERT pre-training and then introduce a stack of efficiency optimizations, which is named Fast-HuBERT in this paper. The proposed Fast-HuBERT can be trained in 1.1 days with 8 V100 GPUs on the Librispeech 960h benchmark, without performance degradation, resulting in a 5.2x speedup, compared to the original implementation. Moreover, we explore two well-studied techniques in the Fast-HuBERT and demonstrate consistent improvements as reported in previous work.

View on arXivView PDF

Code Implementations(1)

yanghaha0908/FastHuBERTCommunity100%
MIT

Official implementation for Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning

1004Shell, PythonJun 30, 20231 years agoMIT
efficiency-optimizationself-supervised-learningspeech-recognition

Similar Papers

Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute

90% match2023

Efficiency-oriented approaches for self-supervised speech representation learning

89% match2023

FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning

87% match1 repo192022

Sustainable self-supervised learning for speech representations

86% match2024

Ara-Best-RQ: Multi Dialectal Arabic SSL

86% match2026

Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget

85% match2024

Cite this paper

@article{yang2023fasthubert,
  title  = {Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning},
  author = {Guanrou Yang and Ziyang Ma and Zhisheng Zheng and Yakun Song and Zhikang Niu and Xie Chen},
  year   = {2023},
  eprint = {2309.13860},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.13860v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.13860.svg)](https://codeofpaper.com/paper/2309.13860)

Discussion