RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations

Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, Atticus Geiger
2/27/2024
cs.CLcs.LG

Official code: explanare/ravel (58 stars, MIT, last commit 2025-10-30).

Abstract

Individual neurons participate in the representation of multiple high-level concepts. To what extent can different interpretability methods successfully disentangle these roles? To help address this question, we introduce RAVEL (Resolving Attribute-Value Entanglements in Language Models), a dataset that enables tightly controlled, quantitative comparisons between a variety of existing interpretability methods. We use the resulting conceptual framework to define the new method of Multi-task Distributed Alignment Search (MDAS), which allows us to find distributed representations satisfying multiple causal criteria. With Llama2-7B as the target language model, MDAS achieves state-of-the-art results on RAVEL, demonstrating the importance of going beyond neuron-level analyses to identify features distributed across activations. We release our benchmark at https://github.com/explanare/ravel.

View on arXivView PDF

Code Implementations(1)

explanare/ravelOfficial100%
MIT

Evaluate interpretability methods on localizing and disentangling concepts in LLMs.

5810Python, Jupyter NotebookFeb 17, 202411 months agoMIT
causal-interventiondisentangled-representationsinterpretabilityinterventionprobing+1 more

Similar Papers

Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future Directions

86% match2026

MIB: A Mechanistic Interpretability Benchmark

84% match1 repo262026

CausalGym: Benchmarking causal interpretability methods on linguistic tasks

83% match1 repo572024

Atlas-Alignment: Making Interpretability Transferable Across Language Models

83% match2025

Interpretability at Scale: Identifying Causal Mechanisms in Alpaca

82% match1 repo8992026

Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network

82% match1 repo42024

Cite this paper

@article{huang2024ravel,
  title  = {RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations},
  author = {Jing Huang and Zhengxuan Wu and Christopher Potts and Mor Geva and Atticus Geiger},
  year   = {2024},
  eprint = {2402.17700},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.17700v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.17700.svg)](https://codeofpaper.com/paper/2402.17700)

Discussion