PSST! Prosodic Speech Segmentation with Transformers

Nathan Roll, Calbert Graham, Simon Todd
2/3/2023
cs.CLcs.SDeess.AS

No official code found. Best implementation: Nathan-Roll1/PSST (28 stars, MIT, last commit 2024-02-25).

Abstract

Self-attention mechanisms have enabled transformers to achieve superhuman-level performance on many speech-to-text (STT) tasks, yet the challenge of automatic prosodic segmentation has remained unsolved. In this paper we finetune Whisper, a pretrained STT model, to annotate intonation unit (IU) boundaries by repurposing low-frequency tokens. Our approach achieves an accuracy of 95.8%, outperforming previous methods without the need for large-scale labeled data or enterprise grade compute resources. We also diminish input signals by applying a series of filters, finding that low pass filters at a 3.2 kHz level improve segmentation performance in out of sample and out of distribution contexts. We release our model as both a transcription tool and a baseline for further improvements in prosodic segmentation.

View on arXivView PDF

Code Implementations(1)

Nathan-Roll1/PSSTCommunity100%
MIT

Prosodic Speech Segmentation with Transformers

285Python, Jupyter NotebookJan 20, 20232 years agoMIT

Similar Papers

Prosody-to-Text: Predicting text from low-pass filtered speech

83% match2026

XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words

81% match2023

Improving TTS for Shanghainese: Addressing Tone Sandhi via Word Segmentation

80% match1 repo282023

Steering intonational tunes in text-to-speech synthesis

80% match1 repo2026

Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment

80% match2024

TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context

79% match2026

Cite this paper

@article{roll2023psst,
  title  = {PSST! Prosodic Speech Segmentation with Transformers},
  author = {Nathan Roll and Calbert Graham and Simon Todd},
  year   = {2023},
  eprint = {2302.01984},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.01984v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.01984.svg)](https://codeofpaper.com/paper/2302.01984)

Discussion