Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach

Maxime Poli, Emmanuel Chemla, Emmanuel Dupoux
9/16/2024
cs.CLcs.SDeess.AS

No official code found. Best implementation: bootphon/spokenlm-phoneme (14 stars, MIT, last commit 2024-10-30).

Abstract

Recent progress in Spoken Language Modeling has shown that learning language directly from speech is feasible. Generating speech through a pipeline that operates at the text level typically loses nuances, intonations, and non-verbal vocalizations. Modeling directly from speech opens up the path to more natural and expressive systems. On the other hand, speech-only systems require up to three orders of magnitude more data to catch up to their text-based counterparts in terms of their semantic abilities. We show that fine-tuning speech representation models on phoneme classification leads to more context-invariant representations, and language models trained on these units achieve comparable lexical comprehension to ones trained on hundred times more data.

View on arXivView PDF

Code Implementations(1)

bootphon/spokenlm-phonemeCommunity100%
MIT

Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach

140Shell, PythonOct 30, 20241 years agoMIT

Similar Papers

From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes

86% match1 repo2024

LAST: Language Model Aware Speech Tokenization

86% match2024

On The Landscape of Spoken Language Models: A Comprehensive Survey

85% match2025

Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech data

85% match2025

SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks

85% match2024

Spoken Language Understanding on Unseen Tasks With In-Context Learning

84% match2025

Cite this paper

@article{poli2024improving,
  title  = {Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach},
  author = {Maxime Poli and Emmanuel Chemla and Emmanuel Dupoux},
  year   = {2024},
  eprint = {2410.00025},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.00025v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.00025.svg)](https://codeofpaper.com/paper/2410.00025)

Discussion