Neural Entropy

Akhil Premkumar
9/5/2024
cs.LGcond-mat.stat-mechcs.IT

No confident code implementation found for this paper. 5 possible matches can be shown below.

Abstract

We explore the connection between deep learning and information theory through the paradigm of diffusion models. A diffusion model converts noise into structured data by reinstating, imperfectly, information that is erased when data was diffused to noise. This information is stored in a neural network during training. We quantify this information by introducing a measure called neural entropy, which is related to the total entropy produced by diffusion. Neural entropy is a function of not just the data distribution, but also the diffusive process itself. Measurements of neural entropy on a few simple image diffusion models reveal that they are extremely efficient at compressing large ensembles of structured data.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 5 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

TENDE: Transfer Entropy Neural Diffusion Estimation

86% match2025

Information Plane Analysis Visualization in Deep Learning via Transfer Entropy

85% match2024

High-entropy Advantage in Neural Networks' Generalizability

84% match2025

Neural Network Diffusion

84% match1 repo8862024

Laplacian-guided Entropy Model in Neural Codec with Blur-dissipated Synthesis

84% match2024

The Information Dynamics of Generative Diffusion

83% match2025

Cite this paper

@article{premkumar2024neural,
  title  = {Neural Entropy},
  author = {Akhil Premkumar},
  year   = {2024},
  eprint = {2409.03817},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.03817v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.03817.svg)](https://codeofpaper.com/paper/2409.03817)

Discussion