Speaker and Language Change Detection using Wav2vec2 and Whisper

Tijn Berns, Nik Vaessen, David A. van Leeuwen
2/18/2023
eess.AScs.CLcs.LG

No confident code implementation found for this paper.

Abstract

We investigate recent transformer networks pre-trained for automatic speech recognition for their ability to detect speaker and language changes in speech. We do this by simply adding speaker (change) or language targets to the labels. For Wav2vec2 pre-trained networks, we also investigate if the representation for the speaker change symbol can be conditioned to capture speaker identity characteristics. Using a number of constructed data sets we show that these capabilities are definitely there, with speaker recognition equal error rates of the order of 10% and language detection error rates of a few percent. We will publish the code for reproducibility.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Evaluating the Effectiveness of Transformer Layers in Wav2Vec 2.0, XLS-R, and Whisper for Speaker Identification Tasks

83% match1 repo32025

Gujarati-English Code-Switching Speech Recognition using ensemble prediction of spoken language

81% match2024

Exploring Spoken Language Identification Strategies for Automatic Transcription of Multilingual Broadcast and Institutional Speech

80% match2024

Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages

80% match2024

Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding

80% match2024

Streaming Speaker Change Detection and Gender Classification for Transducer-Based Multi-Talker Speech Translation

79% match2025

Cite this paper

@article{berns2023speaker,
  title  = {Speaker and Language Change Detection using Wav2vec2 and Whisper},
  author = {Tijn Berns and Nik Vaessen and David A. van Leeuwen},
  year   = {2023},
  eprint = {2302.09381},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.09381v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.09381.svg)](https://codeofpaper.com/paper/2302.09381)

Discussion