DistilXLSR: A Light Weight Cross-Lingual Speech Representation Model

Haoyu Wang, Siyuan Wang, Wei-Qiang Zhang, Jinfeng Bai
6/2/2023
cs.CLcs.SDeess.AS

No official code found. Best implementation: backspacetg/distilXLSR (13 stars, last commit 2025-03-30).

Abstract

Multilingual self-supervised speech representation models have greatly enhanced the speech recognition performance for low-resource languages, and the compression of these huge models has also become a crucial prerequisite for their industrial application. In this paper, we propose DistilXLSR, a distilled cross-lingual speech representation model. By randomly shuffling the phonemes of existing speech, we reduce the linguistic information and distill cross-lingual models using only English data. We also design a layer-jumping initialization method to fully leverage the teacher's pre-trained weights. Experiments on 2 kinds of teacher models and 15 low-resource languages show that our method can reduce the parameters by 50% while maintaining cross-lingual representation ability. Our method is proven to be generalizable to various languages/teacher models and has the potential to improve the cross-lingual performance of the English pre-trained models.

View on arXivView PDF

Code Implementations(1)

backspacetg/distilXLSRCommunity100%

Models and codes for INTERSPEECH 2023 paper DistilXLSR: A Light Weight Cross-Lingual Speech Representation Model

132Shell, PythonMay 27, 20231 years ago

Similar Papers

Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching

87% match2023

Reduce, Reuse, Recycle: Is Perturbed Data better than Other Language augmentation for Low Resource Self-Supervised Speech Models

85% match2023

Supervised Acoustic Embeddings And Their Transferability Across Languages

85% match2023

AuRA: Internalizing Audio Understanding into LLMs as LoRA

84% match2026

CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders

84% match2023

Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages

84% match2023

Cite this paper

@article{wang2023distilxlsr,
  title  = {DistilXLSR: A Light Weight Cross-Lingual Speech Representation Model},
  author = {Haoyu Wang and Siyuan Wang and Wei-Qiang Zhang and Jinfeng Bai},
  year   = {2023},
  eprint = {2306.01303},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.01303v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.01303.svg)](https://codeofpaper.com/paper/2306.01303)

Discussion