MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information

Jianrong Wang, Yuchen Huo, Li Liu, Tianyi Xu, Qi Li +1 more
6/4/2023
cs.SDcs.CV

Official code: SpringHuo/MAVD (20 stars, last commit 2024-04-22).

Abstract

Audio-visual speech recognition (AVSR) gains increasing attention from researchers as an important part of human-computer interaction. However, the existing available Mandarin audio-visual datasets are limited and lack the depth information. To address this issue, this work establishes the MAVD, a new large-scale Mandarin multimodal corpus comprising 12,484 utterances spoken by 64 native Chinese speakers. To ensure the dataset covers diverse real-world scenarios, a pipeline for cleaning and filtering the raw text material has been developed to create a well-balanced reading material. In particular, the latest data acquisition device of Microsoft, Azure Kinect is used to capture depth information in addition to the traditional audio signals and RGB images during data acquisition. We also provide a baseline experiment, which could be used to evaluate the effectiveness of the dataset. The dataset and code will be released at https://github.com/SpringHuo/MAVD.

View on arXivView PDF

Code Implementations(1)

SpringHuo/MAVDOfficial100%

The MAVD represents Mandarin Audio-Visual dataset with Depth information. MAVD has a rich variety of modal data, including audio, RGB images and depth images, etc.

200May 31, 20232 years ago

Similar Papers

BIG-C: a Multimodal Multi-Purpose Dataset for Bemba

85% match1 repo112023

MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation

85% match1 repo4062023

SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition

85% match1 repo82024

LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild

85% match1 repo92023

WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research

85% match1 repo2622023

The NPU-ASLP System Description for Visual Speech Recognition in CNVSRC 2024

84% match2024

Cite this paper

@article{wang2023mavd,
  title  = {MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information},
  author = {Jianrong Wang and Yuchen Huo and Li Liu and Tianyi Xu and Qi Li and Sen Li},
  year   = {2023},
  eprint = {2306.02263},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.02263v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.02263.svg)](https://codeofpaper.com/paper/2306.02263)

Discussion