Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty

Inar Timiryasov, Jean-Loup Tastet
8/3/2023
cs.CL

No official code found. Best implementation: timinar/BabyLlama (84 stars, MIT, last commit 2023-10-18).

Abstract

We present our submission to the BabyLM challenge, whose goal was to improve the sample efficiency of language models. We trained an ensemble consisting of a GPT-2 and small LLaMA models on the developmentally-plausible, 10M-word BabyLM dataset, then distilled it into a small, 58M-parameter LLaMA model, which exceeds in performance both of its teachers as well as a similar model trained without distillation. This suggests that distillation can not only retain the full performance of the teacher model when the latter is trained on a sufficiently small dataset; it can exceed it, and lead to significantly better performance than direct training.

View on arXivView PDF

Code Implementations(1)

timinar/BabyLlamaCommunity100%
MIT

Training code for Baby-Llama, our submission to the strict-small track of the BabyLM challenge.

8413Python, Jupyter NotebookAug 9, 20233 years agoMIT

Similar Papers

Curriculum Learning-Guided Progressive Distillation in Large Language Models

90% match2026

Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation

88% match2023

An Empirical Study of Knowledge Distillation for Code Understanding Tasks

88% match2025

Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection

88% match2023

MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillation

88% match2024

Education distillation:getting student models to learn in shcools

88% match2023

Cite this paper

@article{timiryasov2023baby,
  title  = {Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty},
  author = {Inar Timiryasov and Jean-Loup Tastet},
  year   = {2023},
  eprint = {2308.02019},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.02019v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.02019.svg)](https://codeofpaper.com/paper/2308.02019)

Discussion