Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers

Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis
2/10/2026

No official code found. Best implementation: Sta8is/FUTURIST (48 stars, MIT, last commit 2025-08-19).

Abstract

Semantic future prediction is important for autonomous systems navigating dynamic environments. This paper introduces FUTURIST, a method for multimodal future semantic prediction that uses a unified and efficient visual sequence transformer architecture. Our approach incorporates a multimodal masked visual modeling objective and a novel masking mechanism designed for multimodal training. This allows the model to effectively integrate visible information from various modalities, improving prediction accuracy. Additionally, we propose a VAE-free hierarchical tokenization process, which reduces computational complexity, streamlines the training pipeline, and enables end-to-end training with high-resolution, multimodal inputs. We validate FUTURIST on the Cityscapes dataset, demonstrating state-of-the-art performance in future semantic segmentation for both short- and mid-term forecasting. We provide the implementation code and model weights at https://github.com/Sta8is/FUTURIST.

View on arXivView PDF

Code Implementations(1)

Sta8is/FUTURISTCommunity100%
MIT

[CVPR 2025] Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers

485Python, Jupyter NotebookJan 1, 20251 years agoMIT
future-predictionmasked-transformer

Similar Papers

Emu3: Next-Token Prediction is All You Need

86% match1 repo2.4k2024

ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting

83% match1 repo42025

Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting

83% match2 repos1402026

OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models

83% match1 repo1262025

ForecastOcc: Vision-based Semantic Occupancy Forecasting

83% match2026

Emu3.5: Native Multimodal Models are World Learners

82% match1 repo1.6k2025

Cite this paper

@article{karypidis2026advancing,
  title  = {Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers},
  author = {Efstathios Karypidis and Ioannis Kakogeorgiou and Spyros Gidaris and Nikos Komodakis},
  year   = {2026},
  eprint = {2501.08303},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2501.08303},
  journal = {CVPR 2025 2025}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2501.08303.svg)](https://codeofpaper.com/paper/2501.08303)

Discussion