The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depth

Victor Quétu, Zhu Liao, Enzo Tartaglione
4/27/2024
cs.LG

Official code: VGCQ/EASIER (2 stars, MIT, last commit 2024-12-17).

Abstract

While deep neural networks are highly effective at solving complex tasks, large pre-trained models are commonly employed even to solve consistently simpler downstream tasks, which do not necessarily require a large model's complexity. Motivated by the awareness of the ever-growing AI environmental impact, we propose an efficiency strategy that leverages prior knowledge transferred by large models. Simple but effective, we propose a method relying on an Entropy-bASed Importance mEtRic (EASIER) to reduce the depth of over-parametrized deep neural networks, which alleviates their computational burden. We assess the effectiveness of our method on traditional image classification setups. Our code is available at https://github.com/VGCQ/EASIER.

View on arXivView PDF

Code Implementations(1)

VGCQ/EASIEROfficial100%
MIT

[ECML PKDD 24] The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks’ Depth

20PythonJun 3, 20241 years agoMIT
compressiondeep-learningefficiencypruning

Similar Papers

NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer

84% match2024

Entropic Score metric: Decoupling Topology and Size in Training-free NAS

80% match2023

High-Fidelity Pruning for Large Language Models

80% match1 repo82026

VL2Lite: Task-Specific Knowledge Distillation from Large Vision-Language Models to Lightweight Networks

80% match1 repo202026

Opening the Black Box: predicting the trainability of deep neural networks with reconstruction entropy

80% match1 repo52024

Differentiable Entropy Regularization: A Complexity-Aware Approach for Neural Optimization

80% match2025

Cite this paper

@article{qutu2024simpler,
  title  = {The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depth},
  author = {Victor Quétu and Zhu Liao and Enzo Tartaglione},
  year   = {2024},
  eprint = {2404.18949},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.18949v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.18949.svg)](https://codeofpaper.com/paper/2404.18949)

Discussion