Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters

Euiin Yi, Taehyeon Kim, Hongseok Jeung, Du-Seong Chang, Se-Young Yun
6/24/2024
cs.CL

No official code found. Best implementation: Kthyeon/Multilingual-SpecBench (11 stars, MIT, last commit 2024-07-01).

Abstract

Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. However, the deployment of these models is constrained by high inference time in multilingual settings. To mitigate this challenge, this paper explores a training recipe of an assistant model in speculative decoding, which is leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup in inference time compared to the previous methods. We validate these models across various languages in inference time, out-of-domain speedup, and GPT-4o evaluation.

View on arXivView PDF

Code Implementations(1)

Kthyeon/Multilingual-SpecBenchCommunity100%
MIT

Evaluation of speculative inference over multilingual tasks

110C, RustJun 24, 20242 years agoMIT

Similar Papers

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding

92% match1 repo1.3k2024

Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding

91% match2024

Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies

91% match2025

Accelerating Production LLMs with Combined Token/Embedding Speculators

91% match1 repo2862024

Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration

90% match1 repo82024

Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning

90% match1 repo232026

Cite this paper

@article{yi2024towards,
  title  = {Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters},
  author = {Euiin Yi and Taehyeon Kim and Hongseok Jeung and Du-Seong Chang and Se-Young Yun},
  year   = {2024},
  eprint = {2406.16758},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.16758v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.16758.svg)](https://codeofpaper.com/paper/2406.16758)

Discussion