MViTv2: Improved Multiscale Vision Transformers for Classification and Detection

Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong +2 more
12/2/2021
cs.CV
transformer-attention

Official code: facebookresearch/mvit (456 stars, Apache-2.0, last commit 2024-11-26). Most starred: facebookresearch/detectron2, a community implementation (34,934 stars, Apache-2.0, last commit 2026-10-20, 77× the official stars). 2 more implementations below.

Abstract

In this paper, we study Multiscale Vision Transformers (MViTv2) as a unified architecture for image and video classification, as well as object detection. We present an improved version of MViT that incorporates decomposed relative positional embeddings and residual pooling connections. We instantiate this architecture in five sizes and evaluate it for ImageNet classification, COCO detection and Kinetics video recognition where it outperforms prior work. We further compare MViTv2s' pooling attention to window attention mechanisms where it outperforms the latter in accuracy/compute. Without bells-and-whistles, MViTv2 has state-of-the-art performance in 3 domains: 88.8% accuracy on ImageNet classification, 58.7 boxAP on COCO object detection as well as 86.1% on Kinetics-400 video classification. Code and models are available at https://github.com/facebookresearch/mvit.

View on arXivView PDF

Code Implementations(3)

facebookresearch/mvitOfficial100%
Apache-2.0

Code Release for MViTv2 on Image Recognition.

45652Shell, PythonDec 21, 20211 years agoApache-2.0
facebookresearch/detectron2Community100%
Apache-2.0

Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.

34,9347,942C++, CudaSep 5, 20194 days agoApache-2.0
facebookresearch/SlowFastCommunity100%
Apache-2.0

PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

7,4361,296Shell, PythonAug 20, 20197 months agoApache-2.0

Similar Papers

PosMLP-Video: Spatial and Temporal Relative Position Encoding for Efficient Video Recognition

84% match1 repo42024

Is Space-Time Attention All You Need for Video Understanding?

83% match1 repo1.9k2021

VideoMamba: Spatio-Temporal Selective State Space Model

82% match1 repo272024

Machine Learning for Brain Disorders: Transformers and Visual Transformers

81% match2023

Pyramidal Convolution: Rethinking Convolutional Neural Networks for Visual Recognition

81% match1 repo3302020

DenseNet: Implementing Efficient ConvNet Descriptor Pyramids

81% match1 repo52014

Cite this paper

@article{li2021mvitv,
  title  = {MViTv2: Improved Multiscale Vision Transformers for Classification and Detection},
  author = {Yanghao Li and Chao-Yuan Wu and Haoqi Fan and Karttikeya Mangalam and Bo Xiong and Jitendra Malik and Christoph Feichtenhofer},
  year   = {2021},
  eprint = {2112.01526},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2112.01526}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2112.01526.svg)](https://codeofpaper.com/paper/2112.01526)

Discussion