Don't Throw Away Data: Better Sequence Knowledge Distillation

Jun Wang, Eleftheria Briakou, Hamid Dadkhahi, Rishabh Agarwal, Colin Cherry +1 more
7/15/2024
cs.CL

No confident code implementation found for this paper.

Abstract

A critical component in knowledge distillation is the means of coupling the teacher and student. The predominant sequence knowledge distillation method involves supervised learning of the student against teacher-decoded outputs, and is exemplified by the current state of the art, which incorporates minimum Bayes risk (MBR) decoding. In this paper we seek to integrate MBR more tightly in distillation training, specifically by using several high scoring MBR translations, rather than a single selected sequence, thus capturing a rich diversity of teacher outputs. Our experiments on English to German and English to Japanese translation show consistent improvements over strong baseline methods for both tasks and with varying model sizes. Additionally, we conduct a detailed analysis focusing on data efficiency and capacity curse aspects to elucidate MBR-n and explore its further potential.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages

86% match2025

On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

84% match2023

Memorization Dynamics in Knowledge Distillation for Language Models

84% match2026

SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models

84% match2024

Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Framework

83% match1 repo12025

Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation

83% match2025

Cite this paper

@article{wang2024dont,
  title  = {Don't Throw Away Data: Better Sequence Knowledge Distillation},
  author = {Jun Wang and Eleftheria Briakou and Hamid Dadkhahi and Rishabh Agarwal and Colin Cherry and Trevor Cohn},
  year   = {2024},
  eprint = {2407.10456},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.10456v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.10456.svg)](https://codeofpaper.com/paper/2407.10456)

Discussion