CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning

Medha Hira, Arnav Goel, Anubha Gupta
5/23/2024
cs.CLcs.SDeess.AS

No confident code implementation found for this paper.

Abstract

This paper presents CrossVoice, a novel cascade-based Speech-to-Speech Translation (S2ST) system employing advanced ASR, MT, and TTS technologies with cross-lingual prosody preservation through transfer learning. We conducted comprehensive experiments comparing CrossVoice with direct-S2ST systems, showing improved BLEU scores on tasks such as Fisher Es-En, VoxPopuli Fr-En and prosody preservation on benchmark datasets CVSS-T and IndicTTS. With an average mean opinion score of 3.75 out of 4, speech synthesized by CrossVoice closely rivals human speech on the benchmark, highlighting the efficacy of cascade-based systems and transfer learning in multilingual S2ST with prosody transfer.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Multilingual Prosody Transfer: Comparing Supervised & Transfer Learning

84% match2024

Direct Speech to Speech Translation: A Review

82% match2025

A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation

81% match2023

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

80% match1 repo1642024

Cascaded Cross-Modal Transformer for Audio-Textual Classification

80% match1 repo32024

Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech

80% match2023

Cite this paper

@article{hira2024crossvoice,
  title  = {CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning},
  author = {Medha Hira and Arnav Goel and Anubha Gupta},
  year   = {2024},
  eprint = {2406.00021},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.00021v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.00021.svg)](https://codeofpaper.com/paper/2406.00021)

Discussion