Sequential Diffusion Language Models

Yangzhou Liu, Yue Cao, Hao Li, Gen Luo, Zhe Chen +11 more
9/28/2025
cs.CLcs.LG

Official code: OpenGVLab/SDLM (98 stars, MIT, last commit 2025-12-27).

Abstract

Diffusion language models (DLMs) have strong theoretical efficiency but are limited by fixed-length decoding and incompatibility with key-value (KV) caches. Block diffusion mitigates these issues, yet still enforces a fixed block size and requires expensive training. We introduce Next Sequence Prediction (NSP), which unifies next-token and next-block prediction, enabling the model to adaptively determine the generation length at each step. When the length is fixed to 1, NSP reduces to standard next-token prediction. Building on NSP, we propose Sequential Diffusion Language Model (SDLM), which can retrofit pre-trained autoregressive language models (ALMs) at minimal cost. Specifically, SDLM performs diffusion inference within fixed-size mask blocks, but dynamically decodes consecutive subsequences based on model confidence, thereby preserving KV-cache compatibility and improving robustness to varying uncertainty and semantics across the sequence. Experiments show that SDLM matches or surpasses strong autoregressive baselines using only 3.5M training samples, while achieving 2.1 higher throughput than Qwen-2.5. Notably, the SDLM-32B model delivers even more pronounced efficiency gains, demonstrating the strong scalability potential of our modeling paradigm. Project page and codes: https://github.com/OpenGVLab/SDLM

View on arXivView PDF

Code Implementations(1)

OpenGVLab/SDLMOfficial100%
MIT

Sequential Diffusion Language Model (SDLM) enhances pre-trained autoregressive language models by adaptively determining generation length and maintaining KV-cache compatibility, achieving high efficiency and throughput.

984Shell, PythonAug 25, 20259 months agoMIT
diffusion-modelsgptlanguage-modelllm

Similar Papers

FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion

88% match2025

Diffusion Language Models Know the Answer Before Decoding

88% match1 repo602025

Diffusion Language Models Know the Answer Before Decoding

88% match1 repo602025

Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

88% match1 repo1.0k2025

Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding

88% match1 repo1.1k2025

From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs

88% match1 repo82025

Cite this paper

@article{liu2025sequential,
  title  = {Sequential Diffusion Language Models},
  author = {Yangzhou Liu and Yue Cao and Hao Li and Gen Luo and Zhe Chen and Weiyun Wang and Xiaobo Liang and Biqing Qi and Lijun Wu and Changyao Tian and Yanting Zhang and Yuqiang Li and Tong Lu and Yu Qiao and Jifeng Dai and Wenhai Wang},
  year   = {2025},
  eprint = {2509.24007},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2509.24007v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2509.24007.svg)](https://codeofpaper.com/paper/2509.24007)

Discussion