ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks

Valentin Pelloin, Franck Dary, Nicolas Herve, Benoit Favre, Nathalie Camelin +2 more
7/5/2022
cs.CL

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

We aim at improving spoken language modeling (LM) using very large amount of automatically transcribed speech. We leverage the INA (French National Audiovisual Institute) collection and obtain 19GB of text after applying ASR on 350,000 hours of diverse TV shows. From this, spoken language models are trained either by fine-tuning an existing LM (FlauBERT) or through training a LM from scratch. New models (FlauBERT-Oral) are shared with the community and evaluated for 3 downstream tasks: spoken language understanding, classification of TV shows and speech syntactic parsing. Results show that FlauBERT-Oral can be beneficial compared to its initial FlauBERT version demonstrating that, despite its inherent noisy nature, ASR-generated text can be used to build spoken language models.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Textually Pretrained Speech Language Models

89% match1 repo222026

Spanish Pre-trained BERT Model and Evaluation Data

88% match2 repos5072023

WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words

88% match1 repo192023

Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings

88% match2023

Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data

87% match1 repo10.0k2023

AudioPaLM: A Large Language Model That Can Speak and Listen

87% match2023

Cite this paper

@article{pelloin2022asrgenerated,
  title  = {ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks},
  author = {Valentin Pelloin and Franck Dary and Nicolas Herve and Benoit Favre and Nathalie Camelin and Antoine Laurent and Laurent Besacier},
  year   = {2022},
  eprint = {2207.01893},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2207.01893v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2207.01893.svg)](https://codeofpaper.com/paper/2207.01893)

Discussion