No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations

Walter Simoncini, Spyros Gidaris, Andrei Bursuc, Yuki M. Asano
7/15/2024
cs.CVcs.CLcs.LG

No official code found. Best implementation: WalterSimoncini/fungivision (40 stars, MIT, last commit 2024-10-31). 1 more below.

Abstract

This paper introduces FUNGI, Features from UNsupervised GradIents, a method to enhance the features of transformer encoders by leveraging self-supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. These gradients are projected to a lower dimension and then concatenated with the model's output embedding. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio. Across backbones spanning various sizes and pretraining strategies, FUNGI features provide consistent performance improvements over the embeddings. We also show that using FUNGI features can benefit linear classification, clustering and image retrieval, and that they significantly improve the retrieval-based in-context scene understanding abilities of pretrained models, for example improving upon DINO by +17% for semantic segmentation - without any training.

View on arXivView PDF

Code Implementations(2)

WalterSimoncini/fungivisionCommunity100%
MIT

Library implementation of "No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations"

400Python, Jupyter NotebookJul 5, 20241 years agoMIT
computer-visiondeep-learningretrievalself-supervised-learning
WalterSimoncini/no-train-all-gainCommunity68%
MIT

Code for the paper "No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations"

110Shell, PythonJul 11, 20241 years agoMIT

Similar Papers

Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models

83% match1 repo112024

Gradient Sparsification For Masked Fine-Tuning of Transformers

83% match2023

Investigating Self-Supervised Methods for Label-Efficient Learning

83% match2024

ExpPoint-MAE: Better interpretability and performance for self-supervised point cloud transformers

82% match1 repo132023

Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Crops

82% match2025

Minimizing Energy Costs in Deep Learning Model Training: The Gaussian Sampling Approach

81% match2024

Cite this paper

@article{simoncini2024train,
  title  = {No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations},
  author = {Walter Simoncini and Spyros Gidaris and Andrei Bursuc and Yuki M. Asano},
  year   = {2024},
  eprint = {2407.10964},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.10964v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.10964.svg)](https://codeofpaper.com/paper/2407.10964)

Discussion