Towards Human-like Multimodal Conversational Agent by Generating Engaging Speech

Taesoo Kim, Yongsik Jo, Hyunmin Song, Taehwan Kim
9/18/2025
cs.HCcs.AIcs.CL

Official code: kimtaesu24/MSenC (6 stars, last commit 2025-10-07).

Abstract

Human conversation involves language, speech, and visual cues, with each medium providing complementary information. For instance, speech conveys a vibe or tone not fully captured by text alone. While multimodal LLMs focus on generating text responses from diverse inputs, less attention has been paid to generating natural and engaging speech. We propose a human-like agent that generates speech responses based on conversation mood and responsive style information. To achieve this, we build a novel MultiSensory Conversation dataset focused on speech to enable agents to generate natural speech. We then propose a multimodal LLM-based model for generating text responses and voice descriptions, which are used to generate speech covering paralinguistic information. Experimental results demonstrate the effectiveness of utilizing both visual and audio modalities in conversation to generate engaging speech. The source code is available in https://github.com/kimtaesu24/MSenC

View on arXivView PDF

Code Implementations(1)

kimtaesu24/MSenCOfficial100%

[INTERSPEECH'25] Official repository for "Towards Human-like Multimodal Conversational Agent by Generating Engaging Speech"

62Python, Jupyter NotebookJun 10, 20251 years ago

Similar Papers

Beyond Words: Multimodal LLM Knows When to Speak

83% match2025

Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction

83% match1 repo512024

Developing Enhanced Conversational Agents for Social Virtual Worlds

83% match2025

SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems

82% match1 repo872024

Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialogues

82% match1 repo62025

TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation

82% match2025

Cite this paper

@article{kim2025towards,
  title  = {Towards Human-like Multimodal Conversational Agent by Generating Engaging Speech},
  author = {Taesoo Kim and Yongsik Jo and Hyunmin Song and Taehwan Kim},
  year   = {2025},
  eprint = {2509.14627},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2509.14627v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2509.14627.svg)](https://codeofpaper.com/paper/2509.14627)

Discussion