A Data-Based Perspective on Transfer Learning

Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong, Sung Min Park +1 more
7/12/2022
cs.LG

Official code: MadryLab/data-transfer (35 stars, MIT, last commit 2022-09-23).

Abstract

It is commonly believed that in transfer learning including more pre-training data translates into better performance. However, recent evidence suggests that removing data from the source dataset can actually help too. In this work, we take a closer look at the role of the source dataset's composition in transfer learning and present a framework for probing its impact on downstream performance. Our framework gives rise to new capabilities such as pinpointing transfer learning brittleness as well as detecting pathologies such as data-leakage and the presence of misleading examples in the source dataset. In particular, we demonstrate that removing detrimental datapoints identified by our framework improves transfer learning performance from ImageNet on a variety of target tasks. Code is available at https://github.com/MadryLab/data-transfer

View on arXivView PDF

Code Implementations(1)

MadryLab/data-transferOfficial100%
MIT
351PythonJul 12, 20224 years agoMIT

Similar Papers

When does Bias Transfer in Transfer Learning?

86% match1 repo152022

Bayesian Transfer Learning

81% match2023

The Role of Pre-training Data in Transfer Learning

81% match1 repo102023

Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning

81% match2024

Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning

81% match1 repo142026

Classifier Transfer with Data Selection Strategies for Online Support Vector Machine Classification with Class Imbalance

81% match2022

Cite this paper

@article{jain2022databased,
  title  = {A Data-Based Perspective on Transfer Learning},
  author = {Saachi Jain and Hadi Salman and Alaa Khaddaj and Eric Wong and Sung Min Park and Aleksander Madry},
  year   = {2022},
  eprint = {2207.05739},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2207.05739v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2207.05739.svg)](https://codeofpaper.com/paper/2207.05739)

Discussion