Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers

Zihan Qiu, Zeyu Huang, Youcheng Huang, Jie Fu
2/19/2024
cs.CL

Official code: qiuzh20/Tuning-keys-v.s.-values (4 stars, MIT, last commit 2024-02-20).

Abstract

The feed-forward networks (FFNs) in transformers are recognized as a group of key-value neural memories to restore abstract high-level knowledge. In this work, we conduct an empirical ablation study on updating keys (the 1st layer in the FFNs layer) or values (the 2nd layer in the FFNs layer). We compare those two methods in various knowledge editing and fine-tuning tasks of large language models to draw insights to understand FFNs further. Code is available at $\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\,repo}$.

View on arXivView PDF

Code Implementations(1)

qiuzh20/Tuning-keys-v.s.-valuesOfficial100%
MIT

Official PyTorch Implementation of Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers [Tiny Paper @ ICLR 2024]

40PythonFeb 19, 20242 years agoMIT

Similar Papers

Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models

84% match2025

Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models

83% match2025

A Study on ReLU and Softmax in Transformer

83% match2023

Attractor Patch Networks: Reducing Catastrophic Forgetting with Routed Low-Rank Patch Experts

82% match2026

MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers

82% match2026

Extended Mind Transformers

81% match1 repo1302024

Cite this paper

@article{qiu2024empirical,
  title  = {Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers},
  author = {Zihan Qiu and Zeyu Huang and Youcheng Huang and Jie Fu},
  year   = {2024},
  eprint = {2402.12233},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.12233v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.12233.svg)](https://codeofpaper.com/paper/2402.12233)

Discussion