Learning Individual Styles of Conversational Gesture

Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens +1 more
6/10/2019
cs.CVcs.LGeess.AS

Official code: amirbar/speech2gesture (394 stars, last commit 2024-03-06).

Abstract

Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-modal translation from "in-the-wild'' monologue speech of a single speaker to their hand and arm motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and speech, we release a large video dataset of person-specific gestures. The project website with video, code and data can be found at http://people.eecs.berkeley.edu/~shiry/speech2gesture .

View on arXivView PDF

Code Implementations(1)

amirbar/speech2gestureOfficial100%

code for training the models from the paper "Learning Individual Styles of Conversational Gestures"

39443PythonApr 3, 20192 years ago

Similar Papers

Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement

89% match1 repo12025

Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech?

89% match2026

Understanding Co-speech Gestures in-the-wild

88% match1 repo242025

Towards Unified Co-Speech Gesture Generation via Hierarchical Implicit Periodicity Learning

88% match2025

A Comprehensive Review of Data-Driven Co-Speech Gesture Generation

87% match2023

Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model

87% match1 repo1342024

Cite this paper

@article{ginosar2019learning,
  title  = {Learning Individual Styles of Conversational Gesture},
  author = {Shiry Ginosar and Amir Bar and Gefen Kohavi and Caroline Chan and Andrew Owens and Jitendra Malik},
  year   = {2019},
  eprint = {1906.04160},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1906.04160}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1906.04160.svg)](https://codeofpaper.com/paper/1906.04160)

Discussion