EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning

Jaeyeon Kim, Jaeyoon Jung, Jinjoo Lee, Sang Hoon Woo
1/31/2024
eess.AScs.AIcs.SD

Official code: jaeyeonkim99/EnCLAP (96 stars, MIT, last commit 2024-06-02).

Abstract

We propose EnCLAP, a novel framework for automated audio captioning. EnCLAP employs two acoustic representation models, EnCodec and CLAP, along with a pretrained language model, BART. We also introduce a new training objective called masked codec modeling that improves acoustic awareness of the pretrained language model. Experimental results on AudioCaps and Clotho demonstrate that our model surpasses the performance of baseline models. Source code will be available at https://github.com/jaeyeonkim99/EnCLAP . An online demo is available at https://huggingface.co/spaces/enclap-team/enclap .

View on arXivView PDF

Code Implementations(1)

jaeyeonkim99/EnCLAPOfficial100%
MIT

Official Implementation of EnCLAP (ICASSP 2024)

965Shell, PythonSep 7, 20232 years agoMIT

Similar Papers

DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning

88% match1 repo1.1k2024

AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning

85% match1 repo112024

Classifier-Guided Captioning Across Modalities

85% match1 repo2025

Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding

85% match1 repo102024

LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport

84% match2 repos152025

An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment

84% match1 repo22024

Cite this paper

@article{kim2024enclap,
  title  = {EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning},
  author = {Jaeyeon Kim and Jaeyoon Jung and Jinjoo Lee and Sang Hoon Woo},
  year   = {2024},
  eprint = {2401.17690},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.17690v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.17690.svg)](https://codeofpaper.com/paper/2401.17690)

Discussion