A Closer Look at Spatiotemporal Convolutions for Action Recognition

Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun +1 more
11/30/2017
cs.CV

Official code: facebookresearch/VMZ (1,053 stars, Apache-2.0, last commit 2025-06-17).

Abstract

In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition. Our motivation stems from the observation that 2D CNNs applied to individual frames of the video have remained solid performers in action recognition. In this work we empirically demonstrate the accuracy advantages of 3D CNNs over 2D CNNs within the framework of residual learning. Furthermore, we show that factorizing the 3D convolutional filters into separate spatial and temporal components yields significantly advantages in accuracy. Our empirical study leads to the design of a new spatiotemporal convolutional block "R(2+1)D" which gives rise to CNNs that achieve results comparable or superior to the state-of-the-art on Sports-1M, Kinetics, UCF101 and HMDB51.

View on arXivView PDF

Code Implementations(1)

facebookresearch/VMZOfficial100%
Apache-2.0

VMZ: Model Zoo for Video Modeling

1,053158C++, ShellApr 25, 20181 years agoApache-2.0

Similar Papers

Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification

88% match1 repo1032017

Convolutional Two-Stream Network Fusion for Video Action Recognition

87% match1 repo7152016

F4D: Factorized 4D Convolutional Neural Network for Efficient Video-level Representation Learning

86% match2023

Timeception for Complex Action Recognition

86% match1 repo1562018

Is Space-Time Attention All You Need for Video Understanding?

86% match1 repo1.9k2021

Interpreting video features: a comparison of 3D convolutional networks and convolutional LSTM networks

86% match1 repo112020

Cite this paper

@article{tran2017closer,
  title  = {A Closer Look at Spatiotemporal Convolutions for Action Recognition},
  author = {Du Tran and Heng Wang and Lorenzo Torresani and Jamie Ray and Yann LeCun and Manohar Paluri},
  year   = {2017},
  eprint = {1711.11248},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1711.11248}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1711.11248.svg)](https://codeofpaper.com/paper/1711.11248)

Discussion