The HSIC Bottleneck: Deep Learning without Back-Propagation

Wan-Duo Kurt Ma, J. P. Lewis, W. Bastiaan Kleijn
8/5/2019
cs.LGstat.ML

Official code: choasma/HSIC-bottleneck (94 stars, MIT, PyTorch, last commit 2020-12-13).

Abstract

We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability to learn very deep networks without skip connections. There is no requirement for symmetric feedback or update locking. We find that the HSIC bottleneck provides performance on MNIST/FashionMNIST/CIFAR10 classification comparable to backpropagation with a cross-entropy target, even when the system is not encouraged to make the output resemble the classification labels. Appending a single layer trained with SGD (without backpropagation) to reformat the information further improves performance.

View on arXivView PDF

Code Implementations(1)

choasma/HSIC-bottleneckOfficial100%
PyTorchMIT

The HSIC Bottleneck: Deep Learning without Back-Propagation

9419Shell, PythonDec 3, 20195 years agoMIT
deep-learninghsicinformation-bottleneck-theoryinformation-theorypytorch

Similar Papers

Understanding and Leveraging the Learning Phases of Neural Networks

82% match2023

Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization

81% match1 repo242022

Nonlinear Information Bottleneck

81% match2 repos552017

Justices for Information Bottleneck Theory

81% match2023

Elastic Information Bottleneck

80% match2023

How Does Information Bottleneck Help Deep Learning?

80% match1 repo642023

Cite this paper

@article{ma2019hsic,
  title  = {The HSIC Bottleneck: Deep Learning without Back-Propagation},
  author = {Wan-Duo Kurt Ma and J. P. Lewis and W. Bastiaan Kleijn},
  year   = {2019},
  eprint = {1908.01580},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1908.01580}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1908.01580.svg)](https://codeofpaper.com/paper/1908.01580)

Discussion