SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models

Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai +3 more
7/22/2024
cs.CV

Official code: apple/ml-slowfast-llava (295 stars, last commit 2024-09-16).

Abstract

We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video large language model (LLM) that can jointly capture detailed spatial semantics and long-range temporal context without exceeding the token budget of commonly used LLMs. This is realized by using a two-stream SlowFast design of inputs for Video LLMs to aggregate features from sampled frames in an effective way. Specifically, the Slow pathway extracts features at a low frame rate while keeping as much spatial detail as possible (e.g., with 12x24 tokens), and the Fast pathway operates on a high frame rate but uses a larger spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As a result, this design allows us to adequately capture both spatial and temporal features that are beneficial for detailed video understanding. Experimental results show that SF-LLaVA outperforms existing training-free methods on a wide range of video tasks. On some benchmarks, it achieves comparable or even better performance compared to state-of-the-art Video LLMs that are fine-tuned on video datasets. Code has been made available at: https://github.com/apple/ml-slowfast-llava.

View on arXivView PDF

Code Implementations(1)

apple/ml-slowfast-llavaOfficial100%

SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models

29519Shell, PythonAug 26, 20242 years agoNOASSERTION
multimodal-large-language-modelsvideo-question-answering

Similar Papers

TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability

90% match1 repo1092024

MovieChat+: Question-aware Sparse Memory for Long Video Question Answering

89% match2024

Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs

89% match1 repo602025

VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding

89% match1 repo2932024

Slow-Fast Architecture for Video Multi-Modal Large Language Models

89% match1 repo292025

STORM: Token-Efficient Long Video Understanding for Multimodal LLMs

89% match2025

Cite this paper

@article{xu2024slowfastllava,
  title  = {SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models},
  author = {Mingze Xu and Mingfei Gao and Zhe Gan and Hong-You Chen and Zhengfeng Lai and Haiming Gang and Kai Kang and Afshin Dehghan},
  year   = {2024},
  eprint = {2407.15841},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.15841v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.15841.svg)](https://codeofpaper.com/paper/2407.15841)

Discussion