Exposing Lip-syncing Deepfakes from Mouth Inconsistencies

Soumyya Kanti Datta, Shan Jia, Siwei Lyu
1/18/2024
cs.CV

Official code: skrantidatta/LIPINC (14 stars, last commit 2024-05-21).

Abstract

A lip-syncing deepfake is a digitally manipulated video in which a person's lip movements are created convincingly using AI models to match altered or entirely new audio. Lip-syncing deepfakes are a dangerous type of deepfakes as the artifacts are limited to the lip region and more difficult to discern. In this paper, we describe a novel approach, LIP-syncing detection based on mouth INConsistency (LIPINC), for lip-syncing deepfake detection by identifying temporal inconsistencies in the mouth region. These inconsistencies are seen in the adjacent frames and throughout the video. Our model can successfully capture these irregularities and outperforms the state-of-the-art methods on several benchmark deepfake datasets. Code is available at https://github.com/skrantidatta/LIPINC

View on arXivView PDF

Code Implementations(1)

skrantidatta/LIPINCOfficial100%
142PythonMar 6, 20242 years ago

Similar Papers

Detecting Lip-Syncing Deepfakes: Vision Temporal Transformer for Analyzing Mouth Inconsistencies

95% match1 repo162025

Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes

90% match1 repo1412024

A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild

86% match1 repo13.2k2020

HighSync: High-Quality Lip Synchronization via Latent Diffusion Models

85% match1 repo262026

Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronization

84% match1 repo3792023

PolyGlotFake: A Novel Multilingual and Multimodal DeepFake Dataset

84% match1 repo102024

Cite this paper

@article{datta2024exposing,
  title  = {Exposing Lip-syncing Deepfakes from Mouth Inconsistencies},
  author = {Soumyya Kanti Datta and Shan Jia and Siwei Lyu},
  year   = {2024},
  eprint = {2401.10113},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.10113v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.10113.svg)](https://codeofpaper.com/paper/2401.10113)

Discussion