Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets

Paul Primus, Khaled Koutini, Gerhard Widmer
8/8/2023
eess.AScs.IRcs.LG

No official code found. Best implementation: OptimusPrimus/dcase2023_task6b (5 stars, last commit 2023-06-22).

Abstract

This work presents a text-to-audio-retrieval system based on pre-trained text and spectrogram transformers. Our method projects recordings and textual descriptions into a shared audio-caption space in which related examples from different modalities are close. Through a systematic analysis, we examine how each component of the system influences retrieval performance. As a result, we identify two key components that play a crucial role in driving performance: the self-attention-based audio encoder for audio embedding and the utilization of additional human-generated and synthetic data sets during pre-training. We further experimented with augmenting ClothoV2 captions with available keywords to increase their variety; however, this only led to marginal improvements. Our system ranked first in the 2023's DCASE Challenge, and it outperforms the current state of the art on the ClothoV2 benchmark by 5.6 pp. mAP@10.

View on arXivView PDF

Code Implementations(1)

OptimusPrimus/dcase2023_task6bCommunity100%

CP-JKU's Task6b Submission to DCASE2023

50Shell, PythonMay 15, 20233 years ago

Similar Papers

Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval?

89% match2023

Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval

85% match2026

Parameter Efficient Audio Captioning With Faithful Guidance Using Audio-text Shared Latent Representation

83% match2023

RECAP: Retrieval-Augmented Audio Captioning

82% match1 repo162023

Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning

82% match2023

SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval

82% match2026

Cite this paper

@article{primus2023advancing,
  title  = {Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets},
  author = {Paul Primus and Khaled Koutini and Gerhard Widmer},
  year   = {2023},
  eprint = {2308.04258},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.04258v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.04258.svg)](https://codeofpaper.com/paper/2308.04258)

Discussion