In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning

Yifei Duan, Liu Li, Zirui Zhai, Jinxia Yao
12/17/2024
cs.CL

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

We applied few-shot in-context learning on the OPT-1.3B model for the natural language inference task and employed knowledge distillation to internalize the context information, reducing model parameter from 1.3B to 125M and achieving a size reduction from 2.5GB to 0.25GB. Compared to using in-context learning alone on similarly sized models, this context distillation approach achieved a nearly 50% improvement in out-of-domain accuracy, demonstrating superior knowledge transfer capabilities over prompt-based methods. Furthermore, this approach reduced memory consumption by up to 60% while delivering a 20% improvement in out-of-domain accuracy compared to conventional pattern-based fine-tuning.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Efficient LLM Context Distillation

90% match2024

Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning

86% match2025

Context Tuning for In-Context Optimization

83% match1 repo102025

Fine-Tuned In-Context Learners for Efficient Adaptation

83% match2025

Many-Shot In-Context Learning

83% match4 repos62024

Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science

82% match2024

Cite this paper

@article{duan2024incontext,
  title  = {In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning},
  author = {Yifei Duan and Liu Li and Zirui Zhai and Jinxia Yao},
  year   = {2024},
  eprint = {2412.13243},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.13243v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.13243.svg)](https://codeofpaper.com/paper/2412.13243)

Discussion