WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words

Lukas Wolf, Greta Tuckute, Klemen Kotar, Eghbal Hosseini, Tamar Regev +2 more
12/5/2023
cs.CLcs.AI

No official code found. Best implementation: lu-wo/whisbert (19 stars, MIT, last commit 2024-05-27).

Abstract

Training on multiple modalities of input can augment the capabilities of a language model. Here, we ask whether such a training regime can improve the quality and efficiency of these systems as well. We focus on text--audio and introduce Whisbert, which is inspired by the text--image approach of FLAVA (Singh et al., 2022). In accordance with Babylm guidelines (Warstadt et al., 2023), we pretrain Whisbert on a dataset comprising only 100 million words plus their corresponding speech from the word-aligned version of the People's Speech dataset (Galvez et al., 2021). To assess the impact of multimodality, we compare versions of the model that are trained on text only and on both audio and text simultaneously. We find that while Whisbert is able to perform well on multimodal masked modeling and surpasses the Babylm baselines in most benchmark tasks, it struggles to optimize its complex objective and outperform its text-only Whisbert baseline.

View on arXivView PDF

Code Implementations(1)

lu-wo/whisbertCommunity100%
MIT

babyLM WhisBERT code

190Shell, PythonOct 23, 20232 years agoMIT

Similar Papers

Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings

88% match2023

ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks

88% match2022

ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst

87% match2023

Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study

86% match2026

Trained on 100 million words and still in shape: BERT meets British National Corpus

86% match1 repo342023

Textually Pretrained Speech Language Models

86% match1 repo222026

Cite this paper

@article{wolf2023whisbert,
  title  = {WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words},
  author = {Lukas Wolf and Greta Tuckute and Klemen Kotar and Eghbal Hosseini and Tamar Regev and Ethan Wilcox and Alex Warstadt},
  year   = {2023},
  eprint = {2312.02931},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.02931v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.02931.svg)](https://codeofpaper.com/paper/2312.02931)

Discussion