On-device Streaming Discrete Speech Units

Kwanghee Choi, Masao Someki, Emma Strubell, Shinji Watanabe
6/2/2025
eess.AScs.LGcs.SD

No official code found. Best implementation: Masao-Someki/StreamingDSU (8 stars, last commit 2025-09-03).

Abstract

Discrete speech units (DSUs) are derived from clustering the features of self-supervised speech models (S3Ms). DSUs offer significant advantages for on-device streaming speech applications due to their rich phonetic information, high transmission efficiency, and seamless integration with large language models. However, conventional DSU-based approaches are impractical as they require full-length speech input and computationally expensive S3Ms. In this work, we reduce both the attention window and the model size while preserving the effectiveness of DSUs. Our results demonstrate that we can reduce floating-point operations (FLOPs) by 50% with only a relative increase of 6.5% in character error rate (CER) on the ML-SUPERB 1h dataset. These findings highlight the potential of DSUs for real-time speech processing in resource-constrained environments.

View on arXivView PDF

Code Implementations(1)

Masao-Someki/StreamingDSUCommunity100%

Course project for on-device project

82Shell, PythonSep 26, 20241 years ago

Similar Papers

DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding

85% match2024

Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition

81% match2023

Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Text

81% match2026

Unifying Streaming and Non-streaming Zipformer-based ASR

80% match2025

Discrete Speech Unit Extraction via Independent Component Analysis

80% match1 repo22025

Compact Speech Translation Models via Discrete Speech Units Pretraining

80% match2024

Cite this paper

@article{choi2025ondevice,
  title  = {On-device Streaming Discrete Speech Units},
  author = {Kwanghee Choi and Masao Someki and Emma Strubell and Shinji Watanabe},
  year   = {2025},
  eprint = {2506.01845},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.01845v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.01845.svg)](https://codeofpaper.com/paper/2506.01845)

Discussion