Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Haozhe Zhao +22 more
12/16/2024
cs.CLcs.AIcs.CV

Official code: LMM101/Awesome-Multimodal-Next-Token-Prediction (476 stars, last commit 2025-01-17).

Abstract

Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning tasks across various modalities, achieving considerable success. As Large Language Models (LLMs) have advanced to unify understanding and generation tasks within the textual modality, recent research has shown that tasks from different modalities can also be effectively encapsulated within the NTP framework, transforming the multimodal information into tokens and predict the next one given the context. This survey introduces a comprehensive taxonomy that unifies both understanding and generation within multimodal learning through the lens of NTP. The proposed taxonomy covers five key aspects: Multimodal tokenization, MMNTP model architectures, unified task representation, datasets \& evaluation, and open challenges. This new taxonomy aims to aid researchers in their exploration of multimodal intelligence. An associated GitHub repository collecting the latest papers and repos is available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction

View on arXivView PDF

Code Implementations(1)

LMM101/Awesome-Multimodal-Next-Token-PredictionOfficial100%

[Survey] Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

47613Nov 8, 20241 years ago

Similar Papers

Emu3: Next-Token Prediction is All You Need

88% match1 repo2.4k2024

Pre-Training Curriculum for Multi-Token Prediction in Language Models

88% match1 repo112025

Training LLMs Beyond Next Token Prediction -- Filling the Mutual Information Gap

86% match2025

Better & Faster Large Language Models via Multi-token Prediction

85% match2026

Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction

85% match2024

Predicting the Order of Upcoming Tokens Improves Language Modeling

85% match1 repo482025

Cite this paper

@article{chen2024next,
  title  = {Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey},
  author = {Liang Chen and Zekun Wang and Shuhuai Ren and Lei Li and Haozhe Zhao and Yunshui Li and Zefan Cai and Hongcheng Guo and Lei Zhang and Yizhe Xiong and Yichi Zhang and Ruoyu Wu and Qingxiu Dong and Ge Zhang and Jian Yang and Lingwei Meng and Shujie Hu and Yulong Chen and Junyang Lin and Shuai Bai and Andreas Vlachos and Xu Tan and Minjia Zhang and Wen Xiao and Aaron Yee and Tianyu Liu and Baobao Chang},
  year   = {2024},
  eprint = {2412.18619},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.18619v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.18619.svg)](https://codeofpaper.com/paper/2412.18619)

Discussion