Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing

Alexandra Chronopoulou, Brian Thompson, Prashant Mathur, Yogesh Virkar, Surafel M. Lakew +1 more
2/25/2023
cs.CLcs.SDeess.AS

No official code found. Best implementation: amazon-science/iwslt-autodub-task (21 stars, Apache-2.0, last commit 2024-03-04).

Abstract

Automatic dubbing (AD) is the task of translating the original speech in a video into target language speech. The new target language speech should satisfy isochrony; that is, the new speech should be time aligned with the original video, including mouth movements, pauses, hand gestures, etc. In this paper, we propose training a model that directly optimizes both the translation as well as the speech duration of the generated translations. We show that this system generates speech that better matches the timing of the original speech, compared to prior work, while simplifying the system architecture.

View on arXivView PDF

Code Implementations(1)

amazon-science/iwslt-autodub-taskCommunity100%
Apache-2.0
2111Shell, PythonJan 17, 20232 years agoApache-2.0

Similar Papers

Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters

88% match2023

Audio-Driven Dubbing for User Generated Contents via Style-Aware Semi-Parametric Synthesis

86% match2023

VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models

85% match1 repo172025

TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation

85% match2023

End-to-End Simultaneous Speech Translation with Differentiable Segmentation

84% match1 repo372023

Empowering Video Translation using Multimodal Large Language Models

83% match2026

Cite this paper

@article{chronopoulou2023jointly,
  title  = {Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing},
  author = {Alexandra Chronopoulou and Brian Thompson and Prashant Mathur and Yogesh Virkar and Surafel M. Lakew and Marcello Federico},
  year   = {2023},
  eprint = {2302.12979},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.12979v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.12979.svg)](https://codeofpaper.com/paper/2302.12979)

Discussion