Minimum Description Length Hopfield Networks

Matan Abudy, Nur Lan, Emmanuel Chemla, Roni Katzir
11/11/2023
cs.LGcs.CL

Official code: matanabudy/mdl-hn (4 stars, MIT, last commit 2023-11-16).

Abstract

Associative memory architectures are designed for memorization but also offer, through their retrieval method, a form of generalization to unseen inputs: stored memories can be seen as prototypes from this point of view. Focusing on Modern Hopfield Networks (MHN), we show that a large memorization capacity undermines the generalization opportunity. We offer a solution to better optimize this tradeoff. It relies on Minimum Description Length (MDL) to determine during training which memories to store, as well as how many of them.

View on arXivView PDF

Code Implementations(1)

matanabudy/mdl-hnOfficial100%
MIT

Minimum Description Length Hopfield Networks

40PythonNov 3, 20232 years agoMIT
associative-memoryhopfieldhopfield-networkhopfield-networkshopfield-neural-network+2 more

Similar Papers

Redundancy Maximization as a Principle of Associative Memory Learning

84% match2025

Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes

84% match2024

Modern Hopfield Networks meet Encoded Neural Representations -- Addressing Practical Considerations

83% match2024

Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval

81% match1 repo12024

Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models

81% match1 repo112024

Adaptive Hopfield Network: Rethinking Similarities in Associative Memory

81% match2025

Cite this paper

@article{abudy2023minimum,
  title  = {Minimum Description Length Hopfield Networks},
  author = {Matan Abudy and Nur Lan and Emmanuel Chemla and Roni Katzir},
  year   = {2023},
  eprint = {2311.06518},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.06518v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.06518.svg)](https://codeofpaper.com/paper/2311.06518)

Discussion