Applying sparse autoencoders to unlearn knowledge in language models

Eoin Farrell, Yeu-Tong Lau, Arthur Conmy
10/25/2024
cs.LGcs.AI

No official code found. Best implementation: efarrell1/train_sparse_autoencoder (11 stars, last commit 2024-10-07).

Abstract

We investigate whether sparse autoencoders (SAEs) can be used to remove knowledge from language models. We use the biology subset of the Weapons of Mass Destruction Proxy dataset and test on the gemma-2b-it and gemma-2-2b-it language models. We demonstrate that individual interpretable biology-related SAE features can be used to unlearn a subset of WMDP-Bio questions with minimal side-effects in domains other than biology. Our results suggest that negative scaling of feature activations is necessary and that zero ablating features is ineffective. We find that intervening using multiple SAE features simultaneously can unlearn multiple different topics, but with similar or larger unwanted side-effects than the existing Representation Misdirection for Unlearning technique. Current SAE quality or intervention techniques would need to improve to make SAE-based unlearning comparable to the existing fine-tuning based techniques.

View on arXivView PDF

Code Implementations(1)

efarrell1/train_sparse_autoencoderCommunity100%

Trains Sparse Autoencoders based on outputs from language models

110Python, Jupyter NotebookOct 7, 20242 years ago

Similar Papers

CRISP: Persistent Concept Unlearning via Sparse Autoencoders

86% match1 repo42025

Sparse-Autoencoder-Guided Internal Representation Unlearning for Large Language Models

85% match2025

SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders

85% match2025

A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models

85% match2025

Scaling and evaluating sparse autoencoders

84% match1 repo5972024

Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders

84% match1 repo82025

Cite this paper

@article{farrell2024applying,
  title  = {Applying sparse autoencoders to unlearn knowledge in language models},
  author = {Eoin Farrell and Yeu-Tong Lau and Arthur Conmy},
  year   = {2024},
  eprint = {2410.19278},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.19278v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.19278.svg)](https://codeofpaper.com/paper/2410.19278)

Discussion