Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?

Boris Knyazev, Doha Hwang, Simon Lacoste-Julien
3/7/2023
cs.LGcs.AIcs.CV

No official code found. Best implementation: SamsungSAILMontreal/ghn3 (38 stars, MIT, PyTorch, last commit 2024-08-27).

Abstract

Pretraining a neural network on a large dataset is becoming a cornerstone in machine learning that is within the reach of only a few communities with large-resources. We aim at an ambitious goal of democratizing pretraining. Towards that goal, we train and release a single neural network that can predict high quality ImageNet parameters of other neural networks. By using predicted parameters for initialization we are able to boost training of diverse ImageNet models available in PyTorch. When transferred to other datasets, models initialized with predicted parameters also converge faster and reach competitive final performance.

View on arXivView PDF

Code Implementations(1)

SamsungSAILMontreal/ghn3Community100%
PyTorchMIT

Code for "Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?" [ICML 2023]

3810Shell, PythonMar 7, 20232 years agoMIT
computational-graphsdeep-learninggraphshypernetworksimagenet+3 more

Similar Papers

ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters

85% match1 repo32025

LoGAH: Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameters

85% match1 repo222024

Rethinking generative image pretraining: How far are we from scaling up next-pixel prediction?

83% match2025

Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architectures

83% match2024

Learning to Grow Pretrained Models for Efficient Transformer Training

83% match1 repo922026

ConvNets Match Vision Transformers at Scale

83% match1 repo62023

Cite this paper

@article{knyazev2023scale,
  title  = {Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?},
  author = {Boris Knyazev and Doha Hwang and Simon Lacoste-Julien},
  year   = {2023},
  eprint = {2303.04143},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.04143v2},
  journal = {ICML 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.04143.svg)](https://codeofpaper.com/paper/2303.04143)

Discussion