HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization

Patrick Glandorf, Timo Kaiser, Bodo Rosenhahn
8/14/2023
cs.CV

No official code found. Best implementation: GreenAutoML4FAS/HyperSparse (4 stars, BSD-3-Clause, last commit 2024-09-12).

Abstract

Sparse neural networks are a key factor in developing resource-efficient machine learning applications. We propose the novel and powerful sparse learning method Adaptive Regularized Training (ART) to compress dense into sparse networks. Instead of the commonly used binary mask during training to reduce the number of model weights, we inherently shrink weights close to zero in an iterative manner with increasing weight regularization. Our method compresses the pre-trained model knowledge into the weights of highest magnitude. Therefore, we introduce a novel regularization loss named HyperSparse that exploits the highest weights while conserving the ability of weight exploration. Extensive experiments on CIFAR and TinyImageNet show that our method leads to notable performance gains compared to other sparsification methods, especially in extremely high sparsity regimes up to 99.8 percent model sparsity. Additional investigations provide new insights into the patterns that are encoded in weights with high magnitudes.

View on arXivView PDF

Code Implementations(1)

GreenAutoML4FAS/HyperSparseCommunity100%
BSD-3-Clause

Official implementation of the paper "HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization"

40PythonAug 7, 20232 years agoBSD-3-Clause
efficiencyneural-networkssparsity

Similar Papers

Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning

90% match2024

Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints

87% match2 repos1662022

Accurate Neural Network Pruning Requires Rethinking Sparse Optimization

86% match2023

Sparse Training for Federated Learning with Regularized Error Correction

86% match2023

ELSA: Partial Weight Freezing for Overhead-Free Sparse Network Deployment

86% match2023

Decoupled Weight Decay for Any p Norm

86% match1 repo2024

Cite this paper

@article{glandorf2023hypersparse,
  title  = {HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization},
  author = {Patrick Glandorf and Timo Kaiser and Bodo Rosenhahn},
  year   = {2023},
  eprint = {2308.07163},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.07163v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.07163.svg)](https://codeofpaper.com/paper/2308.07163)

Discussion