Enhancing Accuracy in Deep Learning Using Random Matrix Theory

Leonid Berlyand, Etienne Sandier, Yitzchak Shmalo, Lei Zhang
10/4/2023
cs.LGmath.OC

No confident code implementation found for this paper.

Abstract

We explore the applications of random matrix theory (RMT) in the training of deep neural networks (DNNs), focusing on layer pruning that is reducing the number of DNN parameters (weights). Our numerical results show that this pruning leads to a drastic reduction of parameters while not reducing the accuracy of DNNs and CNNs. Moreover, pruning the fully connected DNNs actually increases the accuracy and decreases the variance for random initializations. Our numerics indicate that this enhancement in accuracy is due to the simplification of the loss landscape. We next provide rigorous mathematical underpinning of these numerical results by proving the RMT-based Pruning Theorem. Our results offer valuable insights into the practical application of RMT for the creation of more efficient and accurate deep-learning models.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Deep Learning Weight Pruning with RMT-SVD: Increasing Accuracy and Reducing Overfitting

92% match1 repo42023

Pruning Deep Neural Networks via a Combination of the Marchenko-Pastur Distribution and Regularization

91% match2025

Random matrix theory and the loss surfaces of neural networks

86% match2023

Rolling the dice for better deep learning performance: A study of randomness techniques in deep neural networks

85% match2024

Neural Redshift: Random Networks are not Random Functions

84% match2024

Neural Network Pruning as Spectrum Preserving Process

84% match2023

Cite this paper

@article{berlyand2023enhancing,
  title  = {Enhancing Accuracy in Deep Learning Using Random Matrix Theory},
  author = {Leonid Berlyand and Etienne Sandier and Yitzchak Shmalo and Lei Zhang},
  year   = {2023},
  eprint = {2310.03165},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.03165v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.03165.svg)](https://codeofpaper.com/paper/2310.03165)

Discussion