Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanism

Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson +3 more
10/25/2023
cs.CLcs.AIcs.LG

No official code found. Best implementation: msakarvadia/AttentionLens (39 stars, MIT, last commit 2024-08-13).

Abstract

Transformer-based Large Language Models (LLMs) are the state-of-the-art for natural language tasks. Recent work has attempted to decode, by reverse engineering the role of linear layers, the internal mechanisms by which LLMs arrive at their final predictions for text completion tasks. Yet little is known about the specific role of attention heads in producing the final token prediction. We propose Attention Lens, a tool that enables researchers to translate the outputs of attention heads into vocabulary tokens via learned attention-head-specific transformations called lenses. Preliminary findings from our trained lenses indicate that attention heads play highly specialized roles in language models. The code for Attention Lens is available at github.com/msakarvadia/AttentionLens.

View on arXivView PDF

Code Implementations(1)

msakarvadia/AttentionLensCommunity100%
MIT

Interpretating the latent space representations of attention head outputs for LLMs

394Shell, PythonJul 3, 20232 years agoMIT

Similar Papers

A-VL: Adaptive Attention for Large Vision-Language Models

88% match1 repo22024

Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation

87% match2025

Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach

87% match2026

Head Pursuit: Probing Attention Specialization in Multimodal Transformers

87% match1 repo112025

Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs

86% match3 repos1162024

EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention

86% match2025

Cite this paper

@article{sakarvadia2023attention,
  title  = {Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanism},
  author = {Mansi Sakarvadia and Arham Khan and Aswathy Ajith and Daniel Grzenda and Nathaniel Hudson and André Bauer and Kyle Chard and Ian Foster},
  year   = {2023},
  eprint = {2310.16270},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.16270v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.16270.svg)](https://codeofpaper.com/paper/2310.16270)

Discussion