Evaluating the Effectiveness of Transformer Layers in Wav2Vec 2.0, XLS-R, and Whisper for Speaker Identification Tasks

Linus Stuhlmann, Michael Alexander Saxer
8/29/2025
cs.SDcs.AIcs.CL

No official code found. Best implementation: slinusc/speaker_identification_evaluation (3 stars, last commit 2025-12-22).

Abstract

This study evaluates the performance of three advanced speech encoder models, Wav2Vec 2.0, XLS-R, and Whisper, in speaker identification tasks. By fine-tuning these models and analyzing their layer-wise representations using SVCCA, k-means clustering, and t-SNE visualizations, we found that Wav2Vec 2.0 and XLS-R capture speaker-specific features effectively in their early layers, with fine-tuning improving stability and performance. Whisper showed better performance in deeper layers. Additionally, we determined the optimal number of transformer layers for each model when fine-tuned for speaker identification tasks.

View on arXivView PDF

Code Implementations(1)

slinusc/speaker_identification_evaluationCommunity75%

Evaluating the Effectiveness of Transformer Layers in Wav2Vec 2.0, XLS-R, and Whisper for Speaker Identification Tasks

31Python, Jupyter NotebookMay 24, 202410 months ago
wav2vecwav2vec2whisperxls-r

Similar Papers

Speaker and Language Change Detection using Wav2vec2 and Whisper

83% match2023

On the Transferability of Whisper-based Representations for "In-the-Wild" Cross-Task Downstream Speech Applications

81% match2023

How Redundant Is the Transformer Stack in Speech Representation Models?

81% match1 repo2024

Normalization through Fine-tuning: Understanding Wav2vec 2.0 Embeddings for Phonetic Analysis

81% match2025

Incorporating Talker Identity Aids With Improving Speech Recognition in Adversarial Environments

80% match2024

Whisper in Focus: Enhancing Stuttered Speech Classification with Encoder Layer Optimization

79% match2023

Cite this paper

@article{stuhlmann2025evaluating,
  title  = {Evaluating the Effectiveness of Transformer Layers in Wav2Vec 2.0, XLS-R, and Whisper for Speaker Identification Tasks},
  author = {Linus Stuhlmann and Michael Alexander Saxer},
  year   = {2025},
  eprint = {2509.00230},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2509.00230v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2509.00230.svg)](https://codeofpaper.com/paper/2509.00230)

Discussion