Transcription and translation of videos using fine-tuned XLSR Wav2Vec2 on custom dataset and mBART

Aniket Tathe, Anand Kamble, Suyash Kumbharkar, Atharva Bhandare, Anirban C. Mitra
3/1/2024
cs.CLcs.CVcs.LG

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

This research addresses the challenge of training an ASR model for personalized voices with minimal data. Utilizing just 14 minutes of custom audio from a YouTube video, we employ Retrieval-Based Voice Conversion (RVC) to create a custom Common Voice 16.0 corpus. Subsequently, a Cross-lingual Self-supervised Representations (XLSR) Wav2Vec2 model is fine-tuned on this dataset. The developed web-based GUI efficiently transcribes and translates input Hindi videos. By integrating XLSR Wav2Vec2 and mBART, the system aligns the translated text with the video timeline, delivering an accessible solution for multilingual video content transcription and translation for personalized voice.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

End to end Hindi to English speech conversion using Bark, mBART and a finetuned XLSR Wav2Vec2

81% match2024

Empowering Low-Resource Language ASR via Large-Scale Pseudo Labeling

80% match2024

GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement

80% match1 repo2032024

Automatic Speech Recognition for Hindi

79% match2024

Enabling ASR for Low-Resource Languages: A Comprehensive Dataset Creation Approach

79% match2024

AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation

79% match1 repo482026

Cite this paper

@article{tathe2024transcription,
  title  = {Transcription and translation of videos using fine-tuned XLSR Wav2Vec2 on custom dataset and mBART},
  author = {Aniket Tathe and Anand Kamble and Suyash Kumbharkar and Atharva Bhandare and Anirban C. Mitra},
  year   = {2024},
  eprint = {2403.00212},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2403.00212v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.00212.svg)](https://codeofpaper.com/paper/2403.00212)

Discussion