Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent +3 more
1/19/2023
cs.CVcs.AIcs.LG

No official code found. Best implementation: facebookresearch/ijepa (3,488 stars, last commit 2024-05-08). 3 more below.

Abstract

This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations. We introduce the Image-based Joint-Embedding Predictive Architecture (I-JEPA), a non-generative approach for self-supervised learning from images. The idea behind I-JEPA is simple: from a single context block, predict the representations of various target blocks in the same image. A core design choice to guide I-JEPA towards producing semantic representations is the masking strategy; specifically, it is crucial to (a) sample target blocks with sufficiently large scale (semantic), and to (b) use a sufficiently informative (spatially distributed) context block. Empirically, when combined with Vision Transformers, we find I-JEPA to be highly scalable. For instance, we train a ViT-Huge/14 on ImageNet using 16 A100 GPUs in under 72 hours to achieve strong downstream performance across a wide range of tasks, from linear classification to object counting and depth prediction.

View on arXivView PDF

Code Implementations(4)

facebookresearch/ijepaCommunity100%

Official codebase for I-JEPA, the Image-based Joint-Embedding Predictive Architecture. First outlined in the CVPR paper, "Self-supervised learning from images with a joint-embedding predictive architecture."

3,488529PythonJun 12, 20232 years agoNOASSERTION
waterdisappear/SAR-JEPACommunity74%

Self-Supervised Learning for SAR ATR with a Joint-Embedding Predictive Architecture

1054C++, CudaFeb 3, 20246 months agoNOASSERTION
automatic-target-recognitionself-supervised-learningsynthetic-aperture-radar
gaasher/I-JEPACommunity70%
MIT

Implementation of I-JEPA from "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture"

29032PythonFeb 3, 20231 years agoMIT
filipbasara0/simple-ijepaCommunity67%
MIT

A simple and efficient implementation of Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA)

57Python, Jupyter NotebookAug 11, 20242 years agoMIT
computer-visioni-jepajepajoint-embeddingrepresentation-learning+2 more

Similar Papers

Enhancing JEPAs with Spatial Conditioning: Robust and Efficient Representation Learning

87% match2024

Learning and Leveraging World Models in Visual Representation Learning

87% match2024

Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning

87% match2024

A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures

86% match1 repo7962026

MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features

85% match2023

S-JEA: Stacked Joint Embedding Architectures for Self-Supervised Visual Representation Learning

84% match2023

Cite this paper

@article{assran2023selfsupervised,
  title  = {Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture},
  author = {Mahmoud Assran and Quentin Duval and Ishan Misra and Piotr Bojanowski and Pascal Vincent and Michael Rabbat and Yann LeCun and Nicolas Ballas},
  year   = {2023},
  eprint = {2301.08243},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2301.08243v3},
  journal = {CVPR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2301.08243.svg)](https://codeofpaper.com/paper/2301.08243)

Discussion