QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition

Abstract

Audiovisual segmentation (AVS) is a challenging task that aims to segment visual objects in videos according to their associated acoustic cues. With multiple sound sources and background disturbances involved, establishing robust correspondences between audio and visual contents poses unique challenges due to (1) complex entanglement across sound sources and (2) frequent changes in the occurrence of distinct sound events. Assuming sound events occur in- dependently, the multi-source semantic space can be rep- resented as the Cartesian product of single-source sub- spaces. We are motivated to decompose the multi-source audio semantics into single-source semantics for more ef- fective interactions with visual content. We propose a se- mantic decomposition method based on product quanti- zation, where the multi-source semantics can be decom- posed and represented by several disentangled and noise- suppressed single-source semantics. Furthermore, we in- troduce a global-to-local quantization mechanism, which distills knowledge from stable global (clip-level) features into local (frame-level) ones, to handle frequent changes in audio semantics. Extensive experiments demonstrate that our semantically decomposed audio representation signifi- cantly improves AVS performance, e.g., +21.2% mIoU on the challenging AVS-Semantic benchmark with ResNet50 backbone.