Joint Audio and Speech Understanding

Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, James Glass
9/25/2023
cs.SDcs.AIeess.AS

No official code found. Best implementation: YuanGongND/ltu (478 stars, last commit 2024-04-24).

Abstract

Humans are surrounded by audio signals that include both speech and non-speech sounds. The recognition and understanding of speech and non-speech audio events, along with a profound comprehension of the relationship between them, constitute fundamental cognitive capabilities. For the first time, we build a machine learning model, called LTU-AS, that has a conceptually similar universal audio perception and advanced reasoning ability. Specifically, by integrating Whisper as a perception module and LLaMA as a reasoning module, LTU-AS can simultaneously recognize and jointly understand spoken text, speech paralinguistics, and non-speech audio events - almost everything perceivable from audio signals.

View on arXivView PDF

Code Implementations(1)

YuanGongND/ltuCommunity100%

Code, Dataset, and Pretrained Models for Audio and Speech Large Language Model "Listen, Think, and Understand".

47841C, C++May 19, 20232 years ago
audioaudio-processingdeep-learninglarge-language-modelsspeech-recognition

Similar Papers

Listen, Think, and Understand

89% match1 repo4782026

SALMONN: Towards Generic Hearing Abilities for Large Language Models

86% match1 repo1.5k2026

Unified Audio Intelligence Without Regressing on Text Intelligence

84% match2026

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities

84% match2026

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

84% match2026

WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations

84% match2025

Cite this paper

@article{gong2023joint,
  title  = {Joint Audio and Speech Understanding},
  author = {Yuan Gong and Alexander H. Liu and Hongyin Luo and Leonid Karlinsky and James Glass},
  year   = {2023},
  eprint = {2309.14405},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.14405v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.14405.svg)](https://codeofpaper.com/paper/2309.14405)

Discussion