Rhythm Modeling for Voice Conversion

Benjamin van Niekerk, Marc-André Carbonneau, Herman Kamper
7/12/2023
eess.AScs.LGcs.SD

Official code: bshall/urhythmic (85 stars, MIT, PyTorch, last commit 2023-08-03).

Abstract

Voice conversion aims to transform source speech into a different target voice. However, typical voice conversion systems do not account for rhythm, which is an important factor in the perception of speaker identity. To bridge this gap, we introduce Urhythmic-an unsupervised method for rhythm conversion that does not require parallel data or text transcriptions. Using self-supervised representations, we first divide source audio into segments approximating sonorants, obstruents, and silences. Then we model rhythm by estimating speaking rate or the duration distribution of each segment type. Finally, we match the target speaking rate or rhythm by time-stretching the speech segments. Experiments show that Urhythmic outperforms existing unsupervised methods in terms of quality and prosody. Code and checkpoints: https://github.com/bshall/urhythmic. Audio demo page: https://ubisoft-laforge.github.io/speech/urhythmic.

View on arXivView PDF

Code Implementations(1)

bshall/urhythmicOfficial
PyTorchMIT

Unsupervised Rhythm Modeling for Voice Conversion

859Python, Jupyter NotebookJun 1, 20233 years agoMIT
pytorchrhythm-conversionrhythm-modelingvoice-conversion

Similar Papers

Unsupervised Rhythm and Voice Conversion of Dysarthric to Healthy Speech for ASR

80% match2025

Speech Rhythm-Based Speaker Embeddings Extraction from Phonemes and Phoneme Duration for Multi-Speaker Speech Synthesis

79% match2024

DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignment

77% match1 repo812024

Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model

77% match2023

ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations

76% match2023

A Unified Model For Voice and Accent Conversion In Speech and Singing using Self-Supervised Learning and Feature Extraction

76% match2024

Cite this paper

@article{niekerk2023rhythm,
  title  = {Rhythm Modeling for Voice Conversion},
  author = {Benjamin van Niekerk and Marc-André Carbonneau and Herman Kamper},
  year   = {2023},
  eprint = {2307.06040},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.06040v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.06040.svg)](https://codeofpaper.com/paper/2307.06040)

Discussion