$E(2)$-Equivariant Vision Transformer

Renjun Xu, Kaifan Yang, Ke Liu, Fengxiang He
6/11/2023
cs.CVcs.AI

Official code: ZJUCDSYangKaifan/GEVit (15 stars, MIT, last commit 2025-02-26).

Abstract

Vision Transformer (ViT) has achieved remarkable performance in computer vision. However, positional encoding in ViT makes it substantially difficult to learn the intrinsic equivariance in data. Initial attempts have been made on designing equivariant ViT but are proved defective in some cases in this paper. To address this issue, we design a Group Equivariant Vision Transformer (GE-ViT) via a novel, effective positional encoding operator. We prove that GE-ViT meets all the theoretical requirements of an equivariant neural network. Comprehensive experiments are conducted on standard benchmark datasets, demonstrating that GE-ViT significantly outperforms non-equivariant self-attention networks. The code is available at https://github.com/ZJUCDSYangKaifan/GEVit.

View on arXivView PDF

Code Implementations(1)

ZJUCDSYangKaifan/GEVitOfficial100%
MIT

This is the GitHub repository for paper " Group Equivariant Vision Transformer".

154PythonMay 27, 20231 years agoMIT

Similar Papers

Reviving Shift Equivariance in Vision Transformers

86% match2023

SO(2) and O(2) Equivariance in Image Recognition with Bessel-Convolutional Neural Networks

83% match1 repo52023

Vision Big Bird: Random Sparsification for Full Attention

83% match2023

Latent Equivariant Operators for Robust Object Recognition: Promise and Challenges

83% match1 repo22026

What Affects Learned Equivariance in Deep Image Recognition Models?

82% match2023

Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP

81% match1 repo82022

Cite this paper

@article{xu2023eequivariant,
  title  = {$E(2)$-Equivariant Vision Transformer},
  author = {Renjun Xu and Kaifan Yang and Ke Liu and Fengxiang He},
  year   = {2023},
  eprint = {2306.06722},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.06722v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.06722.svg)](https://codeofpaper.com/paper/2306.06722)

Discussion