Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models

David Kurzendörfer, Otniel-Bogdan Mercea, A. Sophia Koepke, Zeynep Akata
4/9/2024
cs.CV

Official code: dkurzend/ClipClap-GZSL (24 stars, MIT, last commit 2024-04-15).

Abstract

Audio-visual zero-shot learning methods commonly build on features extracted from pre-trained models, e.g. video or audio classification models. However, existing benchmarks predate the popularization of large multi-modal models, such as CLIP and CLAP. In this work, we explore such large pre-trained models to obtain features, i.e. CLIP for visual features, and CLAP for audio features. Furthermore, the CLIP and CLAP text encoders provide class label embeddings which are combined to boost the performance of the system. We propose a simple yet effective model that only relies on feed-forward neural networks, exploiting the strong generalization capabilities of the new audio, visual and textual features. Our framework achieves state-of-the-art performance on VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL with our new features. Code and data available at: https://github.com/dkurzend/ClipClap-GZSL.

View on arXivView PDF

Code Implementations(1)

dkurzend/ClipClap-GZSLOfficial100%
MIT

Audio-Visual Generalized Zero-Shot Learning using Large Pre-Trained Models

244Shell, PythonDec 2, 20232 years agoMIT
audio-visualaudio-visual-generalized-zero-shot-learningaudio-visual-learningclapclip+5 more

Similar Papers

Audio-visual Generalized Zero-shot Learning the Easy Way

86% match1 repo62026

Music Genre Classification using Large Language Models

84% match2024

CLIP-VAD: Exploiting Vision-Language Models for Voice Activity Detection

84% match2024

DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training

83% match1 repo1452026

Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

83% match2026

Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting

83% match1 repo1252026

Cite this paper

@article{kurzendrfer2024audiovisual,
  title  = {Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models},
  author = {David Kurzendörfer and Otniel-Bogdan Mercea and A. Sophia Koepke and Zeynep Akata},
  year   = {2024},
  eprint = {2404.06309},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.06309v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.06309.svg)](https://codeofpaper.com/paper/2404.06309)

Discussion