How Generative Spoken Language Modeling Encodes Noisy Speech: Investigation from Phonetics to Syntactics

Joonyong Park, Shinnosuke Takamichi, Tomohiko Nakamura, Kentaro Seki, Detai Xin +1 more
6/1/2023
cs.CLcs.AIeess.AS

No confident code implementation found for this paper.

Abstract

We examine the speech modeling potential of generative spoken language modeling (GSLM), which involves using learned symbols derived from data rather than phonemes for speech analysis and synthesis. Since GSLM facilitates textless spoken language processing, exploring its effectiveness is critical for paving the way for novel paradigms in spoken-language processing. This paper presents the findings of GSLM's encoding and decoding effectiveness at the spoken-language and speech levels. Through speech resynthesis experiments, we revealed that resynthesis errors occur at the levels ranging from phonology to syntactics and GSLM frequently resynthesizes natural but content-altered speech.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

AudioPaLM: A Large Language Model That Can Speak and Listen

86% match2023

ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks

86% match2022

Boosting Large Language Model for Speech Synthesis: An Empirical Study

84% match2023

XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs

84% match1 repo942025

Wave to Syntax: Probing spoken language models for syntax

84% match1 repo32023

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

84% match1 repo22.2k2023

Cite this paper

@article{park2023generative,
  title  = {How Generative Spoken Language Modeling Encodes Noisy Speech: Investigation from Phonetics to Syntactics},
  author = {Joonyong Park and Shinnosuke Takamichi and Tomohiko Nakamura and Kentaro Seki and Detai Xin and Hiroshi Saruwatari},
  year   = {2023},
  eprint = {2306.00697},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.00697v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.00697.svg)](https://codeofpaper.com/paper/2306.00697)

Discussion