An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios

Cheng Gong, Erica Cooper, Xin Wang, Chunyu Qiang, Mengzhe Geng +7 more
6/13/2024
cs.CLeess.AS

No official code found. Best implementation: nii-yamagishilab/ZMM-TTS (185 stars, BSD-3-Clause, last commit 2024-03-06).

Abstract

Self-supervised learning (SSL) representations from massively multilingual models offer a promising solution for low-resource language speech tasks. Despite advancements, language adaptation in TTS systems remains an open problem. This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system proposed in our previous work. We conducted experiments on 12 languages using limited data with various fine-tuning configurations. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance. Additionally, we find that the fine-tuning dataset size and number of speakers influence adaptability. Surprisingly, we also observed that using paired data for fine-tuning is not always optimal compared to audio-only data. Beyond speech intelligibility, our analysis covers speaker similarity, language identification, and predicted MOS.

View on arXivView PDF

Code Implementations(1)

nii-yamagishilab/ZMM-TTSCommunity100%
BSD-3-Clause

ZMM-TTS: Zero-shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-supervised Discrete Speech Representations

18511C, AwkNov 8, 20232 years agoBSD-3-Clause

Similar Papers

Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initialization

88% match2024

How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario

87% match2024

ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets

84% match2024

Language Bias in Self-Supervised Learning For Automatic Speech Recognition

84% match2025

Unseen Speaker and Language Adaptation for Lightweight Text-To-Speech with Adapters

84% match2025

Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages

83% match2026

Cite this paper

@article{gong2024initial,
  title  = {An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios},
  author = {Cheng Gong and Erica Cooper and Xin Wang and Chunyu Qiang and Mengzhe Geng and Dan Wells and Longbiao Wang and Jianwu Dang and Marc Tessier and Aidan Pine and Korin Richmond and Junichi Yamagishi},
  year   = {2024},
  eprint = {2406.08911},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.08911v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.08911.svg)](https://codeofpaper.com/paper/2406.08911)

Discussion