Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science

Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu +1 more
7/15/2017
cs.NEcs.AIcs.LG

Official code: dcmocanu/sparse-evolutionary-artificial-neural-networks (264 stars, MIT, last commit 2021-07-21).

Abstract

Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős-Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible.

View on arXivView PDF

Code Implementations(1)

dcmocanu/sparse-evolutionary-artificial-neural-networksOfficial100%
MIT

Always sparse. Never dense. But never say never. A Sparse Training repository for the Adaptive Sparse Connectivity concept and its algorithmic instantiation, i.e. Sparse Evolutionary Training, to boost Deep Learning scalability on various aspects (e.g. memory and computational time efficiency, representation and generalization power).

26464HTML, PythonMar 2, 20185 years agoMIT
adaptive-sparse-connectivityartificial-neural-networksclassificationcomplex-networksdeep-learning+15 more

Similar Papers

A Brain-inspired Algorithm for Training Highly Sparse Neural Networks

88% match1 repo72019

Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware

86% match1 repo2642019

ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks

86% match2023

Rigging the Lottery: Making All Tickets Winners

85% match2 repos3382019

Sparse Networks from Scratch: Faster Training without Losing Performance

84% match1 repo3852019

Robustness in sparse artificial neural networks trained with adaptive topology

83% match2026

Cite this paper

@article{mocanu2017scalable,
  title  = {Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science},
  author = {Decebal Constantin Mocanu and Elena Mocanu and Peter Stone and Phuong H. Nguyen and Madeleine Gibescu and Antonio Liotta},
  year   = {2017},
  eprint = {1707.04780},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1707.04780}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1707.04780.svg)](https://codeofpaper.com/paper/1707.04780)

Discussion