Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition

Joanna Hong, Minsu Kim, Daehun Yoo, Yong Man Ro
7/13/2022
cs.SDcs.AIcs.CV

No official code found. Best implementation: ms-dot-k/AVSR (23 stars, last commit 2024-04-03).

Abstract

This paper focuses on designing a noise-robust end-to-end Audio-Visual Speech Recognition (AVSR) system. To this end, we propose Visual Context-driven Audio Feature Enhancement module (V-CAFE) to enhance the input noisy audio speech with a help of audio-visual correspondence. The proposed V-CAFE is designed to capture the transition of lip movements, namely visual context and to generate a noise reduction mask by considering the obtained visual context. Through context-dependent modeling, the ambiguity in viseme-to-phoneme mapping can be refined for mask generation. The noisy representations are masked out with the noise reduction mask resulting in enhanced audio features. The enhanced audio features are fused with the visual features and taken to an encoder-decoder model composed of Conformer and Transformer for speech recognition. We show the proposed end-to-end AVSR with the V-CAFE can further improve the noise-robustness of AVSR. The effectiveness of the proposed method is evaluated in noisy speech recognition and overlapped speech recognition experiments using the two largest audio-visual datasets, LRS2 and LRS3.

View on arXivView PDF

Code Implementations(1)

ms-dot-k/AVSRCommunity100%

PyTorch implementation of "Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring" (CVPR2023) and "Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition" (Interspeech 2022)

232PythonSep 13, 20232 years agoNOASSERTION

Similar Papers

Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition

86% match1 repo842018

Audio-Visual Efficient Conformer for Robust Speech Recognition

85% match1 repo1012023

Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues

85% match2023

Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

85% match1 repo282023

Visual-Aware Speech Recognition for Noisy Scenarios

85% match2025

SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model

84% match2024

Cite this paper

@article{hong2022visual,
  title  = {Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition},
  author = {Joanna Hong and Minsu Kim and Daehun Yoo and Yong Man Ro},
  year   = {2022},
  eprint = {2207.06020},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2207.06020v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2207.06020.svg)](https://codeofpaper.com/paper/2207.06020)

Discussion