T-VSL: Text-Guided Visual Sound Source Localization in Mixtures

Abstract

Visual sound source localization poses a significant chal-lenge in identifying the semantic region of each sounding source within a video. Existing self-supervised and weakly supervised source localization methods struggle to accu-rately distinguish the semantic regions of each sounding object, particularly in multi-source mixtures. These methods often rely on audio-visual correspondence as guidance, which can lead to substantial performance drops in com-plex multi-source localization scenarios. The lack of access to individual source sounds in multi-source mixtures during training exacerbates the difficulty of learning effective audio-visual correspondence for localization. To ad-dress this limitation, in this paper, we propose incorpo-rating the text modality as an intermediate feature guide using tri-modal joint embedding models (e.g., Audio Clip) to disentangle the semantic audio-visual source correspon-dence in multi-source mixtures. Our framework, dubbed T-VSL, begins by predicting the class of sounding enti-ties in mixtures. Subsequently, the textual representation of each sounding source is employed as guidance to dis-entangle fine-grained audio-visual source correspondence from multi-source mixtures, leveraging the tri-modal Audio-CLIP embedding. This approach enables our framework to handle a flexible number of sources and exhibits promising zero-shot transferability to unseen classes during test time. Extensive experiments conducted on the MUSIC, VG-GSound, and VGGSound-Instruments datasets demonstrate significant performance improvements over state-of-the-art methods. Code is released at https://github.com/enyac-group/T-VSL/tree/main.