Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou +1 more
7/14/2021
eess.AScs.SDeess.IV

Official code: TaoRuijie/TalkNet-ASD (511 stars, MIT, last commit 2023-10-23).

Abstract

Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.

View on arXivView PDF

Code Implementations(1)

TaoRuijie/TalkNet-ASDOfficial100%
MIT

ACM MM 2021: 'Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection'

511102PythonJul 15, 20213 years agoMIT
active-speaker-detectionaudio-visualawesome-asdmultimedia

Similar Papers

LoCoNet: Long-Short Context Network for Active Speaker Detection

87% match2 repos592023

End-To-End Audiovisual Feature Fusion for Active Speaker Detection

85% match2022

Multi-speaker Attention Alignment for Multimodal Social Interaction

83% match2025

MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues

82% match2024

Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction

82% match2022

Multi-Temporal Lip-Audio Memory for Visual Speech Recognition

82% match2023

Cite this paper

@article{tao2021someone,
  title  = {Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection},
  author = {Ruijie Tao and Zexu Pan and Rohan Kumar Das and Xinyuan Qian and Mike Zheng Shou and Haizhou Li},
  year   = {2021},
  eprint = {2107.06592},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2107.06592}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2107.06592.svg)](https://codeofpaper.com/paper/2107.06592)

Discussion