A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTS

Siyang Wang, Gustav Eje Henter, Joakim Gustafson, Éva Székely
3/5/2023
eess.AScs.HCcs.LG

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Recent work has explored using self-supervised learning (SSL) speech representations such as wav2vec2.0 as the representation medium in standard two-stage TTS, in place of conventionally used mel-spectrograms. It is however unclear which speech SSL is the better fit for TTS, and whether or not the performance differs between read and spontaneous TTS, the later of which is arguably more challenging. This study aims at addressing these questions by testing several speech SSLs, including different layers of the same SSL, in two-stage TTS on both read and spontaneous corpora, while maintaining constant TTS model architecture and training settings. Results from listening tests show that the 9th layer of 12-layer wav2vec2.0 (ASR finetuned) outperforms other tested SSLs and mel-spectrogram, in both read and spontaneous TTS. Our work sheds light on both how speech SSL can readily improve current TTS systems, and how SSLs compare in the challenging generative task of TTS. Audio examples can be found at https://www.speech.kth.se/tts-demos/ssr_tts

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesis

90% match2023

What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysis

85% match2024

Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning System

84% match2023

Efficient infusion of self-supervised representations in Automatic Speech Recognition

84% match2024

TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Models

83% match1 repo162025

Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representations

82% match2023

Cite this paper

@article{wang2023comparative,
  title  = {A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTS},
  author = {Siyang Wang and Gustav Eje Henter and Joakim Gustafson and Éva Székely},
  year   = {2023},
  eprint = {2303.02719},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.02719v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.02719.svg)](https://codeofpaper.com/paper/2303.02719)

Discussion