Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation

Yiyang Su, Ali Vosoughi, Shijian Deng, Yapeng Tian, Chenliang Xu
10/18/2023
cs.CVcs.SDeess.AS

No confident code implementation found for this paper.

Abstract

The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view. Current methods struggle with such sounds lacking visible cues. This paper introduces a novel "Audio-Visual Scene-Aware Separation" (AVSA-Sep) framework. It includes a semantic parser for visible and invisible sounds and a separator for scene-informed separation. AVSA-Sep successfully separates both sound types, with joint training and cross-modal alignment enhancing effectiveness.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation

85% match2022

Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics

85% match1 repo92025

High-Quality Visually-Guided Sound Separation from Diverse Categories

84% match1 repo362023

Leveraging Category Information for Single-Frame Visual Sound Source Separation

83% match1 repo62020

Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding

83% match1 repo32023

LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

83% match1 repo192023

Cite this paper

@article{su2023separating,
  title  = {Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation},
  author = {Yiyang Su and Ali Vosoughi and Shijian Deng and Yapeng Tian and Chenliang Xu},
  year   = {2023},
  eprint = {2310.11713},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.11713v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.11713.svg)](https://codeofpaper.com/paper/2310.11713)

Discussion