Word Discovery in Visually Grounded, Self-Supervised Speech Models

Puyuan Peng, David Harwath
3/28/2022
eess.AScs.AIcs.CL

Official code: jasonppy/word-discovery (27 stars, BSD-3-Clause, last commit 2023-12-04). 1 more implementation below.

Abstract

We present a method for visually-grounded spoken term discovery. After training either a HuBERT or wav2vec2.0 model to associate spoken captions with natural images, we show that powerful word segmentation and clustering capability emerges within the model's self-attention heads. Our experiments reveal that this ability is not present to nearly the same extent in the base HuBERT and wav2vec2.0 models, suggesting that the visual grounding task is a crucial component of the word discovery capability we observe. We also evaluate our method on the Buckeye word segmentation and ZeroSpeech spoken term discovery tasks, where we perform on par with or better than currently published methods on several metrics. Code and model weights are available at https://github.com/jasonppy/word-discovery.

View on arXivView PDF

Code Implementations(2)

jasonppy/word-discoveryOfficial100%
BSD-3-Clause

Word Discovery in Visually Grounded, Self-Supervised Speech Models

278Shell, PythonJun 26, 20222 years agoBSD-3-Clause
kamperh/vqwordsegCommunity100%
MIT

Unsupervised phone and word segmentation using dynamic programming on self-supervised VQ features.

398Shell, PythonFeb 22, 20222 years agoMIT

Similar Papers

Connecting Speech to Words through Images

85% match2026

Visually grounded few-shot word acquisition with fewer shots

83% match2023

Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding

83% match2026

Learning to Generate Grounded Visual Captions without Localization Supervision

82% match1 repo462019

Hyperbolic Learning with Synthetic Captions for Open-World Detection

82% match2024

OV-VG: A Benchmark for Open-Vocabulary Visual Grounding

82% match1 repo312023

Cite this paper

@article{peng2022word,
  title  = {Word Discovery in Visually Grounded, Self-Supervised Speech Models},
  author = {Puyuan Peng and David Harwath},
  year   = {2022},
  eprint = {2203.15081},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2203.15081}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2203.15081.svg)](https://codeofpaper.com/paper/2203.15081)

Discussion