Speaker- and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech

Tobias Weise, Philipp Klumpp, Kubilay Can Demir, Paula Andrea Pérez-Toro, Maria Schuster +4 more
7/3/2024
cs.SDcs.AIcs.CL

No official code found. Best implementation: tobwei/APTAI (5 stars, MIT, last commit 2025-01-07).

Abstract

This paper introduces a novel combination of two tasks, previously treated separately: acoustic-to-articulatory speech inversion (AAI) and phoneme-to-articulatory (PTA) motion estimation. We refer to this joint task as acoustic phoneme-to-articulatory speech inversion (APTAI) and explore two different approaches, both working speaker- and text-independently during inference. We use a multi-task learning setup, with the end-to-end goal of taking raw speech as input and estimating the corresponding articulatory movements, phoneme sequence, and phoneme alignment. While both proposed approaches share these same requirements, they differ in their way of achieving phoneme-related predictions: one is based on frame classification, the other on a two-staged training procedure and forced alignment. We reach competitive performance of 0.73 mean correlation for the AAI task and achieve up to approximately 87% frame overlap compared to a state-of-the-art text-dependent phoneme force aligner.

View on arXivView PDF

Code Implementations(1)

tobwei/APTAICommunity100%
MIT

Official repository for Interspeech2024 paper: Speaker- and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech.

50Shell, PythonJun 10, 20241 years agoMIT

Similar Papers

Acoustic to Articulatory Inversion of Speech; Data Driven Approaches, Challenges, Applications, and Future Scope

83% match2025

Training Articulatory Inversion Models for Interspeaker Consistency

83% match2025

Simulating Articulatory Trajectories with Phonological Feature Interpolation

82% match1 repo12024

On the Relationship between Accent Strength and Articulatory Features

79% match2025

TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer

78% match2024

Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI

78% match2024

Cite this paper

@article{weise2024speaker,
  title  = {Speaker- and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech},
  author = {Tobias Weise and Philipp Klumpp and Kubilay Can Demir and Paula Andrea Pérez-Toro and Maria Schuster and Elmar Noeth and Bjoern Heismann and Andreas Maier and Seung Hee Yang},
  year   = {2024},
  eprint = {2407.03132},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.03132v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.03132.svg)](https://codeofpaper.com/paper/2407.03132)

Discussion