A Simple Generalisation of the Implicit Dynamics of In-Context Learning

Francesco Innocenti, El Mehdi Achour
12/12/2025
cs.LGcs.AI

No official code found. Best implementation: francesco-innocenti/generalising-icl (1 star, MIT, last commit 2025-12-15).

Abstract

In-context learning (ICL) refers to the ability of a model to learn new tasks from examples in its input without any parameter updates. In contrast to previous theories of ICL relying on toy models and data settings, recently it has been shown that an abstraction of a transformer block can be seen as implicitly updating the weights of its feedforward network according to the context (Dherin et al., 2025). Here, we provide a simple generalisation of this result for (i) all sequence positions beyond the last, (ii) any transformer block beyond the first, and (iii) more realistic residual blocks including layer normalisation. We empirically verify our theory on simple in-context linear regression tasks and investigate the relationship between the implicit updates related to different tokens within and between blocks. These results help to bring the theory of Dherin et al. (2025) even closer to practice, with potential for validation on large-scale models.

View on arXivView PDF

Code Implementations(1)

francesco-innocenti/generalising-iclCommunity68%
MIT

Code for NeurIPS 2025 workshop paper "A Simple Generalisation of the Implicit Dynamics of In-Context Learning"

10PythonAug 8, 202510 months agoMIT

Similar Papers

Learning without training: The implicit dynamics of in-context learning

87% match1 repo42025

Transformers as Algorithms: Generalization and Stability in In-context Learning

87% match1 repo82023

Re-examining learning linear functions in context

85% match2024

In-Context Learning with Representations: Contextual Generalization of Trained Transformers

85% match2024

Context-Scaling versus Task-Scaling in In-Context Learning

85% match2024

Exact Learning Dynamics of In-Context Learning in Linear Transformers and Its Application to Non-Linear Transformers

85% match2025

Cite this paper

@article{innocenti2025simple,
  title  = {A Simple Generalisation of the Implicit Dynamics of In-Context Learning},
  author = {Francesco Innocenti and El Mehdi Achour},
  year   = {2025},
  eprint = {2512.11255},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2512.11255v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2512.11255.svg)](https://codeofpaper.com/paper/2512.11255)

Discussion