Efficient Adaptation of Multilingual Models for Japanese ASR

Mark Bajo, Haruka Fukukawa, Ryuji Morita, Yuma Ogasawara
12/14/2024
cs.CLcs.AIcs.SD

No official code found. Best implementation: ryujimorita/tokyo_whisperers (9 stars, last commit 2024-12-18).

Abstract

This study explores fine-tuning multilingual ASR (Automatic Speech Recognition) models, specifically OpenAI's Whisper-Tiny, to improve performance in Japanese. While multilingual models like Whisper offer versatility, they often lack precision in specific languages. Conversely, monolingual models like ReazonSpeech excel in language-specific tasks but are less adaptable. Using Japanese-specific datasets and Low-Rank Adaptation (LoRA) along with end-to-end (E2E) training, we fine-tuned Whisper-Tiny to bridge this gap. Our results show that fine-tuning reduced Whisper-Tiny's Character Error Rate (CER) from 32.7 to 20.8 with LoRA and to 14.7 with end-to-end fine-tuning, surpassing Whisper-Base's CER of 20.2. However, challenges with domain-specific terms remain, highlighting the need for specialized datasets. These findings demonstrate that fine-tuning multilingual models can achieve strong language-specific performance while retaining their flexibility. This approach provides a scalable solution for improving ASR in resource-constrained environments and languages with complex writing systems like Japanese.

View on arXivView PDF

Code Implementations(1)

ryujimorita/tokyo_whisperersCommunity100%

This is the final project for Georgia Tech's Deep Learning course (CS 7643). We aim to enhance Japanese Automatic Speech Recognition (ASR) by fine-tuning OpenAI’s Whisper model on Japanese-specific data, assessing whether it can outperform the monolingual ReazonSpeech models in accuracy.

91TeX, ShellNov 14, 20241 years ago

Similar Papers

LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR

87% match2024

Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper

86% match2024

Benchmarking Japanese Speech Recognition on ASR-LLM Setups with Multi-Pass Augmented Generative Error Correction

86% match2024

Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR

86% match2025

Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR

86% match2024

Adaptability of ASR Models on Low-Resource Language: A Comparative Study of Whisper and Wav2Vec-BERT on Bangla

86% match2025

Cite this paper

@article{bajo2024efficient,
  title  = {Efficient Adaptation of Multilingual Models for Japanese ASR},
  author = {Mark Bajo and Haruka Fukukawa and Ryuji Morita and Yuma Ogasawara},
  year   = {2024},
  eprint = {2412.10705},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.10705v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.10705.svg)](https://codeofpaper.com/paper/2412.10705)

Discussion