VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting

Yujin Tang, Peijie Dong, Zhenheng Tang, Xiaowen Chu, Junwei Liang
3/25/2024
cs.CV

Official code: yyyujintang/VMRNN-PyTorch (165 stars, Apache-2.0, last commit 2024-04-09).

Abstract

Combining CNNs or ViTs, with RNNs for spatiotemporal forecasting, has yielded unparalleled results in predicting temporal and spatial dynamics. However, modeling extensive global information remains a formidable challenge; CNNs are limited by their narrow receptive fields, and ViTs struggle with the intensive computational demands of their attention mechanisms. The emergence of recent Mamba-based architectures has been met with enthusiasm for their exceptional long-sequence modeling capabilities, surpassing established vision models in efficiency and accuracy, which motivates us to develop an innovative architecture tailored for spatiotemporal forecasting. In this paper, we propose the VMRNN cell, a new recurrent unit that integrates the strengths of Vision Mamba blocks with LSTM. We construct a network centered on VMRNN cells to tackle spatiotemporal prediction tasks effectively. Our extensive evaluations show that our proposed approach secures competitive results on a variety of tasks while maintaining a smaller model size. Our code is available at https://github.com/yyyujintang/VMRNN-PyTorch.

View on arXivView PDF

Code Implementations(1)

yyyujintang/VMRNN-PyTorchOfficial100%
Apache-2.0

Official repository for CVPR24 Precognition Workshop Paper: VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting.

16512Shell, PythonMar 25, 20242 years agoApache-2.0

Similar Papers

SwinLSTM: Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTM

88% match1 repo1782026

It is all Connected: A New Graph Formulation for Spatio-Temporal Forecasting

84% match2023

Efficient and Effective Time-Series Forecasting with Spiking Neural Networks

83% match1 repo892026

RPMixer: Shaking Up Time Series Forecasting with Random Projections for Large Spatial-Temporal Data

83% match2024

SRVP: Strong Recollection Video Prediction Model Using Attention-Based Spatiotemporal Correlation Fusion

83% match1 repo42025

Transformer with Koopman-Enhanced Graph Convolutional Network for Spatiotemporal Dynamics Forecasting

83% match2025

Cite this paper

@article{tang2024vmrnn,
  title  = {VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting},
  author = {Yujin Tang and Peijie Dong and Zhenheng Tang and Xiaowen Chu and Junwei Liang},
  year   = {2024},
  eprint = {2403.16536},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2403.16536v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.16536.svg)](https://codeofpaper.com/paper/2403.16536)

Discussion