Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining

Takaaki Saeki, Soumi Maiti, Xinjian Li, Shinji Watanabe, Shinnosuke Takamichi +1 more
1/30/2023
eess.AScs.CL

No official code found. Best implementation: Takaaki-Saeki/zm-text-tts (65 stars, Apache-2.0, last commit 2023-05-30).

Abstract

While neural text-to-speech (TTS) has achieved human-like natural synthetic speech, multilingual TTS systems are limited to resource-rich languages due to the need for paired text and studio-quality audio data. This paper proposes a method for zero-shot multilingual TTS using text-only data for the target language. The use of text-only data allows the development of TTS systems for low-resource languages for which only textual resources are available, making TTS accessible to thousands of languages. Inspired by the strong cross-lingual transferability of multilingual language models, our framework first performs masked language model pretraining with multilingual text-only data. Then we train this model with a paired data in a supervised manner, while freezing a language-aware embedding layer. This allows inference even for languages not included in the paired data but present in the text-only data. Evaluation results demonstrate highly intelligible zero-shot TTS with a character error rate of less than 12% for an unseen language.

View on arXivView PDF

Code Implementations(1)

Takaaki-Saeki/zm-text-ttsCommunity100%
Apache-2.0

[IJCAI'23] Learning to Speak from Text for Low-Resource TTS

654Perl, ShellMay 9, 20233 years agoApache-2.0

Similar Papers

Multilingual Text-to-Speech Synthesis for Turkic Languages Using Transliteration

88% match1 repo842023

Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling

87% match2023

A multilingual training strategy for low resource Text to Speech

87% match2024

OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models

86% match2026

Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations

86% match1 repo362025

Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation

86% match1 repo312023

Cite this paper

@article{saeki2023learning,
  title  = {Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining},
  author = {Takaaki Saeki and Soumi Maiti and Xinjian Li and Shinji Watanabe and Shinnosuke Takamichi and Hiroshi Saruwatari},
  year   = {2023},
  eprint = {2301.12596},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2301.12596v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2301.12596.svg)](https://codeofpaper.com/paper/2301.12596)

Discussion