Transformers in Unsupervised Structure-from-Motion

Hemang Chawla, Arnav Varma, Elahe Arani, Bahram Zonooz
12/16/2023
cs.CVcs.AI

No official code found. Best implementation: NeurAI-Lab/MT-SfMLearner (14 stars, MIT, last commit 2023-11-12).

Abstract

Transformers have revolutionized deep learning based computer vision with improved performance as well as robustness to natural corruptions and adversarial attacks. Transformers are used predominantly for 2D vision tasks, including image classification, semantic segmentation, and object detection. However, robots and advanced driver assistance systems also require 3D scene understanding for decision making by extracting structure-from-motion (SfM). We propose a robust transformer-based monocular SfM method that learns to predict monocular pixel-wise depth, ego vehicle's translation and rotation, as well as camera's focal length and principal point, simultaneously. With experiments on KITTI and DDAD datasets, we demonstrate how to adapt different vision transformers and compare them against contemporary CNN-based methods. Our study shows that transformer-based architecture, though lower in run-time efficiency, achieves comparable performance while being more robust against natural corruptions, as well as untargeted and targeted attacks.

View on arXivView PDF

Code Implementations(1)

NeurAI-Lab/MT-SfMLearnerCommunity100%
MIT

Official code for 'Transformers in Unsupervised Structure-from-Motion' and 'Transformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsics'

142Python, MakefileJul 3, 20232 years agoMIT
computer-visiondeep-learningego-motion-estimationmachine-learningmonocular-depth-estimation+6 more

Similar Papers

Efficient 3D Object Reconstruction using Visual Transformers

87% match2023

A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships

86% match2024

A Comparative Study of Vision Transformers and CNNs for Few-Shot Rigid Transformation and Fundamental Matrix Estimation

86% match2025

Towards SAR Automatic Target Recognition MultiCategory SAR Image Classification Based on Light Weight Vision Transformer

86% match2024

Optimizing Vision Transformers with Data-Free Knowledge Transfer

86% match2024

TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up

85% match1 repo1.7k2021

Cite this paper

@article{chawla2023transformers,
  title  = {Transformers in Unsupervised Structure-from-Motion},
  author = {Hemang Chawla and Arnav Varma and Elahe Arani and Bahram Zonooz},
  year   = {2023},
  eprint = {2312.10529},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.10529v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.10529.svg)](https://codeofpaper.com/paper/2312.10529)

Discussion