Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection

Koji Inoue, Bing'er Jiang, Erik Ekstedt, Tatsuya Kawahara, Gabriel Skantze
1/10/2024
cs.CLcs.HCcs.SD

No official code found. Best implementation: ErikEkstedt/VoiceActivityProjection (125 stars, MIT, last commit 2024-05-29).

Abstract

A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialogue stereo audio to future voice activities. The VAP model includes contrastive predictive coding (CPC) and self-attention transformers, followed by a cross-attention transformer. We examine the effect of the input context audio length and demonstrate that the proposed system can operate in real-time with CPU settings, with minimal performance degradation.

View on arXivView PDF

Code Implementations(1)

ErikEkstedt/VoiceActivityProjectionCommunity100%
MIT

Voice Activity Projection Models: Self-supervised learning of Turn-taking Events

12521PythonOct 7, 20222 years agoMIT

Similar Papers

Prompt-Guided Turn-Taking Prediction

88% match2025

Voice Activity Projection Model with Multimodal Encoders

86% match1 repo62025

Multilingual Turn-taking Prediction Using Voice Activity Projection

86% match2024

Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection

85% match2024

Triadic Multi-party Voice Activity Projection for Turn-taking in Spoken Dialogue Systems

84% match2025

Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction

82% match1 repo82025

Cite this paper

@article{inoue2024realtime,
  title  = {Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection},
  author = {Koji Inoue and Bing'er Jiang and Erik Ekstedt and Tatsuya Kawahara and Gabriel Skantze},
  year   = {2024},
  eprint = {2401.04868},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.04868v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.04868.svg)](https://codeofpaper.com/paper/2401.04868)

Discussion