Efficient Knowledge Injection in LLMs via Self-Distillation

Kalle Kujanpää, Pekka Marttinen, Harri Valpola, Alexander Ilin
12/19/2024
cs.CLcs.LG

No official code found. Best implementation: kallekku/prompt-distillation (12 stars, MIT, last commit 2025-08-05).

Abstract

In many practical applications, large language models (LLMs) need to acquire new knowledge not present in their pre-training data. Efficiently leveraging this knowledge usually relies on supervised fine-tuning or retrieval-augmented generation (RAG). Although RAG has emerged as the industry standard for knowledge injection, fine-tuning has not yet achieved comparable success. This paper proposes utilizing prompt distillation, a self-distillation-based method previously explored primarily for style alignment and instruction tuning, to internalize new factual knowledge from free-form documents. Unlike prior methods, our approach requires neither larger teacher models nor structured knowledge formats. Across multiple LLM sizes and model families, we show that prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. We analyze the key factors contributing to prompt distillation's effectiveness and examine how it scales.

View on arXivView PDF

Code Implementations(1)

kallekku/prompt-distillationCommunity60%
MIT

Efficient Knowledge Injection in LLMs via Self-Distillation (TMLR)

122PythonAug 5, 20251 years agoMIT

Similar Papers

PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning

87% match1 repo132024

DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation

86% match2025

A Survey on Knowledge Distillation of Large Language Models

86% match1 repo1.3k2024

Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting

86% match1 repo52024

Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models

86% match2025

Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models

86% match2024

Cite this paper

@article{kujanp2024efficient,
  title  = {Efficient Knowledge Injection in LLMs via Self-Distillation},
  author = {Kalle Kujanpää and Pekka Marttinen and Harri Valpola and Alexander Ilin},
  year   = {2024},
  eprint = {2412.14964},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.14964v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.14964.svg)](https://codeofpaper.com/paper/2412.14964)

Discussion