Sample-Efficient Diffusion for Text-To-Speech Synthesis

Justin Lovelace, Soham Ray, Kwangyoun Kim, Kilian Q. Weinberger, Felix Wu
9/1/2024
cs.SDcs.AIcs.LG

No official code found. Best implementation: justinlovelace/SESD (61 stars, MIT, last commit 2024-10-28).

Abstract

This work introduces Sample-Efficient Speech Diffusion (SESD), an algorithm for effective speech synthesis in modest data regimes through latent diffusion. It is based on a novel diffusion architecture, that we call U-Audio Transformer (U-AT), that efficiently scales to long sequences and operates in the latent space of a pre-trained audio autoencoder. Conditioned on character-aware language model representations, SESD achieves impressive results despite training on less than 1k hours of speech - far less than current state-of-the-art systems. In fact, it synthesizes more intelligible speech than the state-of-the-art auto-regressive model, VALL-E, while using less than 2% the training data.

View on arXivView PDF

Code Implementations(1)

justinlovelace/SESDCommunity100%
MIT
613Shell, PythonJun 10, 20241 years agoMIT

Similar Papers

Autoregressive Diffusion Transformer for Text-to-Speech Synthesis

87% match2024

A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI

86% match2023

LatentSpeech: Latent Diffusion for Text-To-Speech Generation

85% match2024

DiffVoice: Text-to-Speech with Latent Diffusion

85% match2023

NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

84% match1 repo1.3k2023

CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model

84% match1 repo2142023

Cite this paper

@article{lovelace2024sampleefficient,
  title  = {Sample-Efficient Diffusion for Text-To-Speech Synthesis},
  author = {Justin Lovelace and Soham Ray and Kwangyoun Kim and Kilian Q. Weinberger and Felix Wu},
  year   = {2024},
  eprint = {2409.03717},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.03717v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.03717.svg)](https://codeofpaper.com/paper/2409.03717)

Discussion