Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localization

Edward Fish, Jon Weinbren, Andrew Gilbert
10/5/2023
cs.CVcs.LGcs.MM

No confident code implementation found for this paper.

Abstract

Temporal Action Localization (TAL) aims to identify actions' start, end, and class labels in untrimmed videos. While recent advancements using transformer networks and Feature Pyramid Networks (FPN) have enhanced visual feature recognition in TAL tasks, less progress has been made in the integration of audio features into such frameworks. This paper introduces the Multi-Resolution Audio-Visual Feature Fusion (MRAV-FF), an innovative method to merge audio-visual data across different temporal resolutions. Central to our approach is a hierarchical gated cross-attention mechanism, which discerningly weighs the importance of audio information at diverse temporal scales. Such a technique not only refines the precision of regression boundaries but also bolsters classification confidence. Importantly, MRAV-FF is versatile, making it compatible with existing FPN TAL architectures and offering a significant enhancement in performance when audio data is available.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

The Solution for Temporal Sound Localisation Task of ICCV 1st Perception Test Challenge 2023

84% match2024

UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization

83% match1 repo332024

AUD-TGN: Advancing Action Unit Detection with Temporal Convolution and GPT-2 in Wild Audiovisual Contexts

83% match2024

CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization

83% match1 repo102024

TIM: A Time Interval Machine for Audio-Visual Action Recognition

82% match1 repo542026

Hierarchical Multi-Stage Transformer Architecture for Context-Aware Temporal Action Localization

82% match2025

Cite this paper

@article{fish2023multiresolution,
  title  = {Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localization},
  author = {Edward Fish and Jon Weinbren and Andrew Gilbert},
  year   = {2023},
  eprint = {2310.03456},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.03456v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.03456.svg)](https://codeofpaper.com/paper/2310.03456)

Discussion