Masked Feature Prediction for Self-Supervised Visual Pre-Training

Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille +1 more
12/16/2021
cs.CVcs.LG

Official code: facebookresearch/SlowFast (7,436 stars, Apache-2.0, last commit 2026-03-16).

Abstract

We present Masked Feature Prediction (MaskFeat) for self-supervised pre-training of video models. Our approach first randomly masks out a portion of the input sequence and then predicts the feature of the masked regions. We study five different types of features and find Histograms of Oriented Gradients (HOG), a hand-crafted feature descriptor, works particularly well in terms of both performance and efficiency. We observe that the local contrast normalization in HOG is essential for good results, which is in line with earlier work using HOG for visual recognition. Our approach can learn abundant visual knowledge and drive large-scale Transformer-based models. Without using extra model weights or supervision, MaskFeat pre-trained on unlabeled videos achieves unprecedented results of 86.7% with MViT-L on Kinetics-400, 88.3% on Kinetics-600, 80.4% on Kinetics-700, 39.8 mAP on AVA, and 75.0% on SSv2. MaskFeat further generalizes to image input, which can be interpreted as a video with a single frame and obtains competitive results on ImageNet.

View on arXivView PDF

Code Implementations(1)

facebookresearch/SlowFastOfficial100%
Apache-2.0

PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

7,4361,296Shell, PythonAug 20, 20197 months agoApache-2.0

Similar Papers

A Large-Scale Analysis on Self-Supervised Video Representation Learning

90% match2023

A Large-Scale Analysis on Contextual Self-Supervised Video Representation Learning

89% match2025

Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning

87% match1 repo1362022

SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning

86% match2 repos162025

E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training

86% match1 repo3052025

BEiT: BERT Pre-Training of Image Transformers

86% match1 repo22.2k2021

Cite this paper

@article{wei2021masked,
  title  = {Masked Feature Prediction for Self-Supervised Visual Pre-Training},
  author = {Chen Wei and Haoqi Fan and Saining Xie and Chao-Yuan Wu and Alan Yuille and Christoph Feichtenhofer},
  year   = {2021},
  eprint = {2112.09133},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2112.09133}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2112.09133.svg)](https://codeofpaper.com/paper/2112.09133)

Discussion