BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition

Alexandros Haliassos, Andreas Zinonos, Rodrigo Mira, Stavros Petridis, Maja Pantic
4/2/2024
cs.CV

No official code found. Best implementation: ahaliassos/raven (82 stars, MIT, last commit 2025-02-27).

Abstract

Self-supervision has recently shown great promise for learning visual and auditory speech representations from unlabelled data. In this work, we propose BRAVEn, an extension to the recent RAVEn method, which learns speech representations entirely from raw audio-visual data. Our modifications to RAVEn enable BRAVEn to achieve state-of-the-art results among self-supervised methods in various settings. Moreover, we observe favourable scaling behaviour by increasing the amount of unlabelled data well beyond other self-supervised works. In particular, we achieve 20.0% / 1.7% word error rate for VSR / ASR on the LRS3 test set, with only 30 hours of labelled data and no external ASR models. Our results suggest that readily available unlabelled audio-visual data can largely replace costly transcribed data.

View on arXivView PDF

Code Implementations(1)

ahaliassos/ravenCommunity100%
MIT

Official implementation of RAVEn (ICLR 2023) and BRAVEn (ICASSP 2024)

829Shell, PythonMar 1, 20231 years agoMIT

Similar Papers

Enhancing Speech Emotion Recognition through Segmental Average Pooling of Self-Supervised Learning Features

86% match2024

AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR

86% match2026

HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition

85% match1 repo1212024

Pre-training with Synthetic Patterns for Audio

85% match2024

Prototype based Masked Audio Model for Self-Supervised Learning of Sound Event Detection

84% match1 repo1062024

Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs

84% match1 repo402026

Cite this paper

@article{haliassos2024braven,
  title  = {BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition},
  author = {Alexandros Haliassos and Andreas Zinonos and Rodrigo Mira and Stavros Petridis and Maja Pantic},
  year   = {2024},
  eprint = {2404.02098},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.02098v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.02098.svg)](https://codeofpaper.com/paper/2404.02098)

Discussion