Synthetic Audio Helps for Cognitive State Tasks

Adil Soubki, John Murzaku, Peter Zeng, Owen Rambow
2/10/2025
cs.SDcs.AIcs.CL

No official code found. Best implementation: adil-soubki/sad-training (2 stars, last commit 2025-02-18).

Abstract

The NLP community has broadly focused on text-only approaches of cognitive state tasks, but audio can provide vital missing cues through prosody. We posit that text-to-speech models learn to track aspects of cognitive state in order to produce naturalistic audio, and that the signal audio models implicitly identify is orthogonal to the information that language models exploit. We present Synthetic Audio Data fine-tuning (SAD), a framework where we show that 7 tasks related to cognitive state modeling benefit from multimodal training on both text and zero-shot synthetic audio data from an off-the-shelf TTS system. We show an improvement over the text-only modality when adding synthetic audio data to text-only corpora. Furthermore, on tasks and corpora that do contain gold audio, we show our SAD framework achieves competitive performance with text and synthetic audio compared to text and gold audio.

View on arXivView PDF

Code Implementations(1)

adil-soubki/sad-trainingCommunity100%

The code and data used to train multimodal models with synthetic audio data, as described in Soubki and Murzaku et al. (2025), which was accepted to NAACL 2025 Findings.

21Shell, PythonAug 30, 20241 years ago

Similar Papers

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

83% match2026

Probing Audio-Generation Capabilities of Text-Based Language Models

83% match2025

From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data

82% match2025

Natural language guidance of high-fidelity text-to-speech with synthetic annotations

81% match2024

Improving semantic understanding in speech language models via brain-tuning

81% match1 repo12026

Enhancing Audio-Language Models through Self-Supervised Post-Training with Text-Audio Pairs

81% match1 repo32024

Cite this paper

@article{soubki2025synthetic,
  title  = {Synthetic Audio Helps for Cognitive State Tasks},
  author = {Adil Soubki and John Murzaku and Peter Zeng and Owen Rambow},
  year   = {2025},
  eprint = {2502.06922},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2502.06922v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2502.06922.svg)](https://codeofpaper.com/paper/2502.06922)

Discussion