Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling

Kazuki Yano, Jun Suzuki, Shinji Watanabe
4/1/2026
cs.CL

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

Adapting pre-trained text Large Language Models (LLMs) into Speech Language Models (Speech LMs) via continual pretraining on speech data is promising, but often degrades the original text capabilities. We propose Multimodal Depth Upscaling, an extension of an emerging strategy in continual LLM pre-training, where new transformer layers are inserted into a frozen text LLM and only the added layers are trained on speech data. Experiments with SmolLM2-360M and SmolLM2-1.7B on 48k hours of English Automatic Speech Recognition (ASR) data show that depth up-scaling achieves ASR comparable to full fine-tuning while causing far less text degradation than both full fine-tuning and Low-Rank Adaptation (LoRA). We further show that incorporating E-Branchformer, an architecture designed for speech recognition, as the inserted layers achieves ASR that matches or surpasses full fine-tuning on the larger model while reducing text degradation by over 75% with 60% fewer trainable parameters.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models

85% match2024

Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment

84% match2024

Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training

83% match2025

When Life gives you LLMs, make LLM-ADE: Large Language Models with Adaptive Data Engineering

83% match2024

Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs

82% match2024

Latent Speech-Text Transformer

82% match2025

Cite this paper

@article{yano2026adapting,
  title  = {Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling},
  author = {Kazuki Yano and Jun Suzuki and Shinji Watanabe},
  year   = {2026},
  eprint = {2604.00489},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2604.00489v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2604.00489.svg)](https://codeofpaper.com/paper/2604.00489)

Discussion