Leveraging Language Model Capabilities for Sound Event Detection

Hualei Wang, Jianguo Mao, Zhifang Guo, Jiarui Wan, Hong Liu +1 more
8/22/2023
cs.SDcs.AIeess.AS

No official code found. Best implementation: lysanderism/SED-LMs (5 stars, last commit 2024-10-23).

Abstract

Large language models reveal deep comprehension and fluent generation in the field of multi-modality. Although significant advancements have been achieved in audio multi-modality, existing methods are rarely leverage language model for sound event detection (SED). In this work, we propose an end-to-end framework for understanding audio features while simultaneously generating sound event and their temporal location. Specifically, we employ pretrained acoustic models to capture discriminative features across different categories and language models for autoregressive text generation. Conventional methods generally struggle to obtain features in pure audio domain for classification. In contrast, our framework utilizes the language model to flexibly understand abundant semantic context aligned with the acoustic representation. The experimental results showcase the effectiveness of proposed method in enhancing timestamps precision and event classification.

View on arXivView PDF

Code Implementations(1)

lysanderism/SED-LMsCommunity68%

Leveraging Language Model Capabilities for Sound Event Detection

50Python, Jupyter NotebookMay 22, 20241 years ago

Similar Papers

Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings

84% match2023

Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding

84% match1 repo342024

Did You Hear That? Introducing AADG: A Framework for Generating Benchmark Data in Audio Anomaly Detection

84% match2024

Listen, Think, and Understand

83% match1 repo4782023

Enhancing Audio-Language Models through Self-Supervised Post-Training with Text-Audio Pairs

83% match1 repo32024

T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining

83% match2024

Cite this paper

@article{wang2023leveraging,
  title  = {Leveraging Language Model Capabilities for Sound Event Detection},
  author = {Hualei Wang and Jianguo Mao and Zhifang Guo and Jiarui Wan and Hong Liu and Xiangdong Wang},
  year   = {2023},
  eprint = {2308.11530},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.11530v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.11530.svg)](https://codeofpaper.com/paper/2308.11530)

Discussion