Towards Comprehensive Semantic Speech Embeddings for Chinese Dialects

Kalvin Chang, Yiwen Shao, Jiahong Li, Dong Yu
1/12/2026
cs.CL

Official code: kalvinchang/yubao (1 star, last commit 2026-01-13).

Abstract

Despite having hundreds of millions of speakers, Chinese dialects lag behind Mandarin in speech and language technologies. Most varieties are primarily spoken, making dialect-to-Mandarin speech-LLMs (large language models) more practical than dialect LLMs. Building dialect-to-Mandarin speech-LLMs requires speech representations with cross-dialect semantic alignment between Chinese dialects and Mandarin. In this paper, we achieve such a cross-dialect semantic alignment by training a speech encoder with ASR (automatic speech recognition)-only data, as demonstrated by speech-to-speech retrieval on a new benchmark of spoken Chinese varieties that we contribute. Our speech encoder further demonstrates state-of-the-art ASR performance on Chinese dialects. Together, our Chinese dialect benchmark, semantically aligned speech representations, and speech-to-speech retrieval evaluation lay the groundwork for future Chinese dialect speech-LLMs. We release the benchmark at https://github.com/kalvinchang/yubao.

View on arXivView PDF

Code Implementations(1)

kalvinchang/yubaoOfficial100%

Speech retrieval benchmark for Chinese varieties

10Shell, PythonDec 19, 20259 months ago

Similar Papers

Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis

87% match2025

Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs

84% match2024

Large Language Model Should Understand Pinyin for Chinese ASR Error Correction

83% match2024

LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generation

83% match2024

Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representation

83% match2024

Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets

83% match1 repo152024

Cite this paper

@article{chang2026towards,
  title  = {Towards Comprehensive Semantic Speech Embeddings for Chinese Dialects},
  author = {Kalvin Chang and Yiwen Shao and Jiahong Li and Dong Yu},
  year   = {2026},
  eprint = {2601.07274},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2601.07274v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2601.07274.svg)](https://codeofpaper.com/paper/2601.07274)

Discussion