Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning

Joy Crosbie, Ekaterina Shutova
7/9/2024
cs.CL

No official code found. Best implementation: JoyC177/Induction_Heads_ICL (5 stars, Apache-2.0, last commit 2025-02-05).

Abstract

Large language models (LLMs) have shown a remarkable ability to learn and perform complex tasks through in-context learning (ICL). However, a comprehensive understanding of its internal mechanisms is still lacking. This paper explores the role of induction heads in a few-shot ICL setting. We analyse two state-of-the-art models, Llama-3-8B and InternLM2-20B on abstract pattern recognition and NLP tasks. Our results show that even a minimal ablation of induction heads leads to ICL performance decreases of up to ~32% for abstract pattern recognition tasks, bringing the performance close to random. For NLP tasks, this ablation substantially decreases the model's ability to benefit from examples, bringing few-shot ICL performance close to that of zero-shot prompts. We further use attention knockout to disable specific induction patterns, and present fine-grained evidence for the role that the induction mechanism plays in ICL.

View on arXivView PDF

Code Implementations(1)

JoyC177/Induction_Heads_ICLCommunity63%
Apache-2.0

Code for the paper: "Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning"

51PythonFeb 5, 20251 years agoApache-2.0

Similar Papers

Which Attention Heads Matter for In-Context Learning?

92% match1 repo102025

Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning

89% match2025

In-Context Learning Without Copying

88% match2025

Identifying Semantic Induction Heads to Understand In-Context Learning

88% match2024

A circuit for predicting hierarchical structure in-context in Large Language Models

87% match2025

In-Context Language Learning: Architectures and Algorithms

87% match1 repo542024

Cite this paper

@article{crosbie2024induction,
  title  = {Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning},
  author = {Joy Crosbie and Ekaterina Shutova},
  year   = {2024},
  eprint = {2407.07011},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.07011v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.07011.svg)](https://codeofpaper.com/paper/2407.07011)

Discussion