XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech

Linh The Nguyen, Thinh Pham, Dat Quoc Nguyen
5/31/2023
cs.CLcs.SDeess.AS

Official code: VinAIResearch/XPhoneBERT (358 stars, MIT, last commit 2024-07-22).

Abstract

We present XPhoneBERT, the first multilingual model pre-trained to learn phoneme representations for the downstream text-to-speech (TTS) task. Our XPhoneBERT has the same model architecture as BERT-base, trained using the RoBERTa pre-training approach on 330M phoneme-level sentences from nearly 100 languages and locales. Experimental results show that employing XPhoneBERT as an input phoneme encoder significantly boosts the performance of a strong neural TTS model in terms of naturalness and prosody and also helps produce fairly high-quality speech with limited training data. We publicly release our pre-trained XPhoneBERT with the hope that it would facilitate future research and downstream TTS applications for multiple languages. Our XPhoneBERT model is available at https://github.com/VinAIResearch/XPhoneBERT

View on arXivView PDF

Code Implementations(1)

VinAIResearch/XPhoneBERTOfficial100%
MIT

XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech (INTERSPEECH 2023)

35843Cython, PythonApr 27, 20232 years agoMIT

Similar Papers

Performance Comparison of Pre-trained Models for Speech-to-Text in Turkish: Whisper-Small and Wav2Vec2-XLS-R-300M

87% match2023

Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation

87% match1 repo312023

ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks

87% match2022

Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding

86% match1 repo22023

Transfer Learning of Transformer-based Speech Recognition Models from Czech to Slovak

86% match2023

Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions

86% match2 repos2692023

Cite this paper

@article{nguyen2023xphonebert,
  title  = {XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech},
  author = {Linh The Nguyen and Thinh Pham and Dat Quoc Nguyen},
  year   = {2023},
  eprint = {2305.19709},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.19709v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.19709.svg)](https://codeofpaper.com/paper/2305.19709)

Discussion