Predicting the Order of Upcoming Tokens Improves Language Modeling

Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji
8/26/2025
cs.LG

Official code: zaydzuhri/token-order-prediction (48 stars, last commit 2025-09-12).

Abstract

Multi-Token Prediction (MTP) has been proposed as an auxiliary objective to improve next-token prediction (NTP) in language model training but shows inconsistent improvements, underperforming in standard NLP benchmarks. We argue that MTP's exact future token prediction is too difficult as an auxiliary loss. Instead, we propose Token Order Prediction (TOP), which trains models to order upcoming tokens by their proximity using a learning-to-rank loss. TOP requires only a single additional unembedding layer compared to MTP's multiple transformer layers. We pretrain models of 340M, 1.8B, and 7B parameters using NTP, MTP, and TOP objectives. Results on eight standard NLP benchmarks show that TOP overall outperforms both NTP and MTP even at scale. Our code is available at https://github.com/zaydzuhri/token-order-prediction

View on arXivView PDF

Code Implementations(1)

zaydzuhri/token-order-predictionOfficial100%

Landing repository for the paper "Predicting the Order of Upcoming Tokens Improves Language Modeling"

483PythonAug 26, 20251 years ago

Similar Papers

Pre-Training Curriculum for Multi-Token Prediction in Language Models

88% match1 repo112025

On multi-token prediction for efficient LLM inference

86% match2025

Better & Faster Large Language Models via Multi-token Prediction

86% match2026

Training LLMs Beyond Next Token Prediction -- Filling the Mutual Information Gap

85% match2025

Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

85% match1 repo4762024

Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Prediction

85% match2025

Cite this paper

@article{zuhri2025predicting,
  title  = {Predicting the Order of Upcoming Tokens Improves Language Modeling},
  author = {Zayd M. K. Zuhri and Erland Hilman Fuadi and Alham Fikri Aji},
  year   = {2025},
  eprint = {2508.19228},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2508.19228v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2508.19228.svg)](https://codeofpaper.com/paper/2508.19228)

Discussion