Retina Vision Transformer (RetinaViT): Introducing Scaled Patches into Vision Transformers

Abstract

Humans see low spatial frequency components before high spatial frequency components. Drawing on this neuroscientific inspiration, we investigate the effect of introducing patches from different spatial frequencies into Vision Transformers (ViTs). We name this model Retina Vision Transformer (RetinaViT) due to its inspiration from the human visual system. Our experiments on benchmark data show that RetinaViT exhibits a strong tendency to attend to low spatial frequency components in the early layers, and shifts its attention to high spatial frequency components as the network goes deeper. This tendency emerged by itself without any additional inductive bias, and aligns with the visual processing order of the human visual system. We hypothesise that RetinaViT captures structural features, or the gist of the scene, in earlier layers, before attending to fine details in subsequent layers, which is the reverse of the processing order of mainstream backbone vision models, such as CNNs. We also observe that RetinaViT is more robust to significant reductions in model size compared to the original ViT, which we hypothesise to have come from its ability to capture the gist of the scene early.