Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning

Nhi Hoai Doan, Tatsuya Hiraoka, Kentaro Inui
7/10/2025
cs.CL

No confident code implementation found for this paper.

Abstract

This paper investigates the relationship between large language models' (LLMs) ability to recognize repetitive input patterns and their performance on in-context learning (ICL). In contrast to prior work that has primarily focused on attention heads, we examine this relationship from the perspective of skill neurons, specifically repetition neurons. Our experiments reveal that the impact of these neurons on ICL performance varies depending on the depth of the layer in which they reside. By comparing the effects of repetition neurons and induction heads, we further identify strategies for reducing repetitive outputs while maintaining strong ICL capabilities.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models

90% match2025

Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning

89% match1 repo52024

Repetitions are not all alike: distinct mechanisms sustain repetition in language models

89% match2025

Which Attention Heads Matter for In-Context Learning?

89% match1 repo102025

Repetition Neurons: How Do Language Models Produce Repetitions?

88% match2024

A circuit for predicting hierarchical structure in-context in Large Language Models

86% match2025

Cite this paper

@article{doan2025understanding,
  title  = {Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning},
  author = {Nhi Hoai Doan and Tatsuya Hiraoka and Kentaro Inui},
  year   = {2025},
  eprint = {2507.07810},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2507.07810v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2507.07810.svg)](https://codeofpaper.com/paper/2507.07810)

Discussion