Length Aware Speech Translation for Video Dubbing

Harveen Singh Chadha, Aswin Shanmugam Subramanian, Vikas Joshi, Shubham Bansal, Jian Xue +2 more
5/31/2025
cs.CLcs.AIcs.SD

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

In video dubbing, aligning translated audio with the source audio is a significant challenge. Our focus is on achieving this efficiently, tailored for real-time, on-device video dubbing scenarios. We developed a phoneme-based end-to-end length-sensitive speech translation (LSST) model, which generates translations of varying lengths short, normal, and long using predefined tags. Additionally, we introduced length-aware beam search (LABS), an efficient approach to generate translations of different lengths in a single decoding pass. This approach maintained comparable BLEU scores compared to a baseline without length awareness while significantly enhancing synchronization quality between source and target audio, achieving a mean opinion score (MOS) gain of 0.34 for Spanish and 0.65 for Korean, respectively.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization

84% match2025

Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning

83% match2024

Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing

82% match1 repo212023

Empowering Video Translation using Multimodal Large Language Models

80% match2026

Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters

80% match2023

Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages

79% match2024

Cite this paper

@article{chadha2025length,
  title  = {Length Aware Speech Translation for Video Dubbing},
  author = {Harveen Singh Chadha and Aswin Shanmugam Subramanian and Vikas Joshi and Shubham Bansal and Jian Xue and Rupeshkumar Mehta and Jinyu Li},
  year   = {2025},
  eprint = {2506.00740},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.00740v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.00740.svg)](https://codeofpaper.com/paper/2506.00740)

Discussion