JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing

Qili Wang, Dajiang Wu, Zihang Xu, Junshi Huang, Jun Lv
1/3/2025
cs.CV

No official code found. Best implementation: JOY-MM/JoyGen (437 stars, Apache-2.0, last commit 2025-02-27).

Abstract

Significant progress has been made in talking-face video generation research; however, precise lip-audio synchronization and high visual quality remain challenging in editing lip shapes based on input audio. This paper introduces JoyGen, a novel two-stage framework for talking-face generation, comprising audio-driven lip motion generation and visual appearance synthesis. In the first stage, a 3D reconstruction model and an audio2motion model predict identity and expression coefficients respectively. Next, by integrating audio features with a facial depth map, we provide comprehensive supervision for precise lip-audio synchronization in facial generation. Additionally, we constructed a Chinese talking-face dataset containing 130 hours of high-quality video. JoyGen is trained on the open-source HDTF dataset and our curated dataset. Experimental results demonstrate superior lip-audio synchronization and visual quality achieved by our method.

View on arXivView PDF

Code Implementations(1)

JOY-MM/JoyGenCommunity100%
Apache-2.0

talking-face video editing

43756Shell, PythonJan 2, 20251 years agoApache-2.0

Similar Papers

Controllable Talking Face Generation by Implicit Facial Keypoints Editing

91% match1 repo342024

S3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis

91% match2026

Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert

91% match1 repo4292026

High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

91% match2024

Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation

91% match2024

Identity-Preserving Talking Face Generation with Landmark and Appearance Priors

90% match3 repos7332026

Cite this paper

@article{wang2025joygen,
  title  = {JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing},
  author = {Qili Wang and Dajiang Wu and Zihang Xu and Junshi Huang and Jun Lv},
  year   = {2025},
  eprint = {2501.01798},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2501.01798v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2501.01798.svg)](https://codeofpaper.com/paper/2501.01798)

Discussion