Low-latency Real-time Voice Conversion on CPU

Konstantine Sadov, Matthew Hutter, Asara Near
11/1/2023
cs.SDcs.LGeess.AS

Official code: KoeAI/LLVC (434 stars, MIT, last commit 2023-11-06).

Abstract

We adapt the architectures of previous audio manipulation and generation neural networks to the task of real-time any-to-one voice conversion. Our resulting model, LLVC ($\textbf{L}$ow-latency $\textbf{L}$ow-resource $\textbf{V}$oice $\textbf{C}$onversion), has a latency of under 20ms at a bitrate of 16kHz and runs nearly 2.8x faster than real-time on a consumer CPU. LLVC uses both a generative adversarial architecture as well as knowledge distillation in order to attain this performance. To our knowledge LLVC achieves both the lowest resource usage as well as the lowest latency of any open-source voice conversion model. We provide open-source samples, code, and pretrained model weights at https://github.com/KoeAI/LLVC.

View on arXivView PDF

Code Implementations(1)

KoeAI/LLVCOfficial
MIT
43445PythonOct 28, 20232 years agoMIT

Similar Papers

StreamVC: Real-Time Low-Latency Voice Conversion

81% match2 repos1282024

ALO-VC: Any-to-any Low-latency One-shot Voice Conversion

81% match2023

Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference

81% match2024

VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction

77% match2025

ChipChat: Low-Latency Cascaded Conversational Agent in MLX

77% match2025

NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference

77% match1 repo92025

Cite this paper

@article{sadov2023lowlatency,
  title  = {Low-latency Real-time Voice Conversion on CPU},
  author = {Konstantine Sadov and Matthew Hutter and Asara Near},
  year   = {2023},
  eprint = {2311.00873},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.00873v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.00873.svg)](https://codeofpaper.com/paper/2311.00873)

Discussion