VoxServe: Streaming-Centric Serving System for Speech Language Models

Keisuke Kamahori, Wei-Tzu Lee, Atindra Jha, Rohan Kadekodi, Stephanie Wang +2 more
1/30/2026
cs.LGcs.AIcs.DC

Official code: vox-serve/vox-serve (78 stars, Apache-2.0, last commit 2026-04-20).

Abstract

Deploying modern Speech Language Models (SpeechLMs) in streaming settings requires systems that provide low latency, high throughput, and strong guarantees of streamability. Existing systems fall short of supporting diverse models flexibly and efficiently. We present VoxServe, a unified serving system for SpeechLMs that optimizes streaming performance. VoxServe introduces a model-execution abstraction that decouples model architecture from system-level optimizations, thereby enabling support for diverse SpeechLM architectures within a single framework. Building on this abstraction, VoxServe implements streaming-aware scheduling and an asynchronous inference pipeline to improve end-to-end efficiency. Evaluations across multiple modern SpeechLMs show that VoxServe achieves 10-20x higher throughput than existing implementations at comparable latency while maintaining high streaming viability. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.

View on arXivView PDF

Code Implementations(1)

vox-serve/vox-serveOfficial100%
Apache-2.0

A Streaming-Native Serving Engine for TTS/STS Models

7810PythonMay 29, 20256 months agoApache-2.0
inferencemachine-learningservingspeechspeech-to-speech+1 more

Similar Papers

Voxtral Realtime

83% match2026

CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

80% match1 repo21.8k2024

VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency

80% match2025

BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5

80% match2024

LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM

80% match1 repo3082025

SpeakStream: Streaming Text-to-Speech with Interleaved Data

80% match2025

Cite this paper

@article{kamahori2026voxserve,
  title  = {VoxServe: Streaming-Centric Serving System for Speech Language Models},
  author = {Keisuke Kamahori and Wei-Tzu Lee and Atindra Jha and Rohan Kadekodi and Stephanie Wang and Arvind Krishnamurthy and Baris Kasikci},
  year   = {2026},
  eprint = {2602.00269},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2602.00269v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2602.00269.svg)](https://codeofpaper.com/paper/2602.00269)

Discussion