Dynamic Depth Decoding: Faster Speculative Decoding for LLMs

Oscar Brown, Zhengjie Wang, Andrea Do, Nikhil Mathew, Cheng Yu
8/30/2024
cs.CLcs.AI

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

The acceleration of Large Language Models (LLMs) with speculative decoding provides a significant runtime improvement without any loss of accuracy. Currently, EAGLE-2 is the state-of-the-art speculative decoding method, improving on EAGLE with a dynamic draft tree. We introduce Dynamic Depth Decoding (DDD), which optimises EAGLE-2's tree drafting method using a dynamic depth. This extends the average speedup that EAGLE-2 achieves over EAGLE by $44\%$, giving DDD an average speedup of $3.16$x.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

SSSD: Simply-Scalable Speculative Decoding

86% match2024

Decoding Speculative Decoding

85% match1 repo162024

Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models

85% match2025

Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models

85% match2024

Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding

85% match2025

Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions

85% match2025

Cite this paper

@article{brown2024dynamic,
  title  = {Dynamic Depth Decoding: Faster Speculative Decoding for LLMs},
  author = {Oscar Brown and Zhengjie Wang and Andrea Do and Nikhil Mathew and Cheng Yu},
  year   = {2024},
  eprint = {2409.00142},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.00142v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.00142.svg)](https://codeofpaper.com/paper/2409.00142)

Discussion