A Whisper transformer for audio captioning trained with synthetic captions and transfer learning

Marek Kadlčík, Adam Hájek, Jürgen Kieslich, Radosław Winiecki
5/15/2023
cs.SDcs.LGeess.AS

No official code found. Best implementation: prompteus/audio-captioning (30 stars, MIT, last commit 2024-12-26).

Abstract

The field of audio captioning has seen significant advancements in recent years, driven by the availability of large-scale audio datasets and advancements in deep learning techniques. In this technical report, we present our approach to audio captioning, focusing on the use of a pretrained speech-to-text Whisper model and pretraining on synthetic captions. We discuss our training procedures and present our experiments' results, which include model size variations, dataset mixtures, and other hyperparameters. Our findings demonstrate the impact of different training strategies on the performance of the audio captioning model. Our code and trained models are publicly available on GitHub and Hugging Face Hub.

View on arXivView PDF

Code Implementations(1)

prompteus/audio-captioningCommunity100%
MIT

Audio captioning - DCASE challenge 2023 task 6a

303HTML, ShellMar 15, 20231 years agoMIT

Similar Papers

Efficient Audio Captioning Transformer with Patchout and Text Guidance

84% match2023

Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding

83% match1 repo102024

On the Transferability of Whisper-based Representations for "In-the-Wild" Cross-Task Downstream Speech Applications

83% match2023

Classifier-Guided Captioning Across Modalities

82% match1 repo2025

Exploring the Role of Audio in Video Captioning

82% match2023

RECAP: Retrieval-Augmented Audio Captioning

82% match1 repo162023

Cite this paper

@article{kadlk2023whisper,
  title  = {A Whisper transformer for audio captioning trained with synthetic captions and transfer learning},
  author = {Marek Kadlčík and Adam Hájek and Jürgen Kieslich and Radosław Winiecki},
  year   = {2023},
  eprint = {2305.09690},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.09690v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.09690.svg)](https://codeofpaper.com/paper/2305.09690)

Discussion