Context Tuning for In-Context Optimization

Jack Lu, Ryan Teehan, Zhenbang Yang, Mengye Ren
7/6/2025
cs.CLcs.AIcs.LG

No official code found. Best implementation: agentic-learning-ai-lab/context-tuning (10 stars, MIT, last commit 2026-02-03).

Abstract

We introduce Context Tuning, a simple and effective method to significantly enhance few-shot adaptation of language models (LLMs) without fine-tuning model parameters. While prompt-based adaptation techniques have demonstrated the effectiveness of lightweight adaptation methods for LLMs, they typically initialize a trainable prompt or prefix with irrelevant tokens for the task at hand. In contrast, Context Tuning initializes the trainable prompt or prefix with task-specific demonstration examples, leveraging the model's inherent In-Context Learning (ICL) ability to extract relevant information for improved few-shot learning performance. Extensive evaluations on benchmarks such as CrossFit, UnifiedQA, MMLU, BIG-Bench Hard, and ARC demonstrate that Context Tuning outperforms traditional prompt-based adaptation methods and achieves competitive accuracy to Test-Time Training with significantly higher training efficiency.

View on arXivView PDF

Code Implementations(1)

agentic-learning-ai-lab/context-tuningCommunity56%
MIT

Public code release for the paper "Context Tuning for In-Context Optimization"

100PythonJul 12, 20258 months agoMIT

Similar Papers

Fine-Tuned In-Context Learners for Efficient Adaptation

91% match2025

Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods

91% match1 repo22024

Efficient LLM Context Distillation

87% match2024

Optimization-Inspired Few-Shot Adaptation for Large Language Models

87% match2025

Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science

87% match2024

Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages

87% match2025

Cite this paper

@article{lu2025context,
  title  = {Context Tuning for In-Context Optimization},
  author = {Jack Lu and Ryan Teehan and Zhenbang Yang and Mengye Ren},
  year   = {2025},
  eprint = {2507.04221},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2507.04221v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2507.04221.svg)](https://codeofpaper.com/paper/2507.04221)

Discussion