Meta Learning Text-to-Speech Synthesis in over 7000 Languages

Florian Lux, Sarina Meyer, Lyonel Behringer, Frank Zalkow, Phat Do +3 more
6/10/2024
cs.CLcs.LGcs.SD

No official code found. Best implementation: DigitalPhonetics/IMS-Toucan (2,211 stars, Apache-2.0, PyTorch, last commit 2026-01-25).

Abstract

In this work, we take on the challenging task of building a single text-to-speech synthesis system that is capable of generating speech in over 7000 languages, many of which lack sufficient data for traditional TTS development. By leveraging a novel integration of massively multilingual pretraining and meta learning to approximate language representations, our approach enables zero-shot speech synthesis in languages without any available data. We validate our system's performance through objective measures and human evaluation across a diverse linguistic landscape. By releasing our code and models publicly, we aim to empower communities with limited linguistic resources and foster further innovation in the field of speech technology.

View on arXivView PDF

Code Implementations(1)

DigitalPhonetics/IMS-ToucanCommunity100%
PyTorchApache-2.0

Controllable and fast Text-to-Speech for over 7000 languages!

2,211319PythonAug 5, 20218 months agoApache-2.0
deep-learningpytorchspeechspeech-processingspeech-synthesis+3 more

Similar Papers

XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model

84% match1 repo452024

MParrotTTS: Multilingual Multi-speaker Text to Speech Synthesis in Low Resource Setting

84% match2023

SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models

83% match2025

Enhancing Code-switched Text-to-Speech Synthesis Capability in Large Language Models with only Monolingual Corpora

83% match2024

A multilingual training strategy for low resource Text to Speech

82% match2024

Instruction Data Generation and Unsupervised Adaptation for Speech Language Models

82% match2024

Cite this paper

@article{lux2024meta,
  title  = {Meta Learning Text-to-Speech Synthesis in over 7000 Languages},
  author = {Florian Lux and Sarina Meyer and Lyonel Behringer and Frank Zalkow and Phat Do and Matt Coler and Emanuël A. P. Habets and Ngoc Thang Vu},
  year   = {2024},
  eprint = {2406.06403},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.06403v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.06403.svg)](https://codeofpaper.com/paper/2406.06403)

Discussion