Acceleration Multiple Heads Decoding for LLM via Dynamic Tree Attention

Zhendong Zhang
2/9/2025
cs.CVcs.CL

No official code found. Best implementation: zzd1992/MEDUSA-Plus (0 stars, MIT, last commit 2025-02-11).

Abstract

Multiple heads decoding accelerates the inference of Large Language Models (LLMs) by predicting next several tokens simultaneously. It generates and verifies multiple candidate sequences in parallel via tree attention with a fixed structure. In this paper, we replace the fixed tree attention with dynamic tree attention on multiple head decoding, specifically in the context of MEDUSA. We propose a simple and low complexity strategy to generate candidates and construct the dynamic tree structure. Preliminary experiments show that the proposed method improves the decoding efficiency of multiple head decoding for LLMs while maintaining the generation quality. This result demonstrates the potential for improvement of multiple head decoding in candidate generation.

View on arXivView PDF

Code Implementations(1)

zzd1992/MEDUSA-PlusCommunity58%
MIT

MEDUSA+: Acceleration Multiple Heads Decoding for LLM via Dynamic Tree Attention

00Python, Jupyter NotebookFeb 9, 20251 years agoMIT

Similar Papers

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

89% match1 repo2.8k2024

Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference

84% match2024

Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models

84% match2025

Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs

84% match2024

When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models

82% match1 repo352024

DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference

82% match1 repo542024

Cite this paper

@article{zhang2025acceleration,
  title  = {Acceleration Multiple Heads Decoding for LLM via Dynamic Tree Attention},
  author = {Zhendong Zhang},
  year   = {2025},
  eprint = {2502.05947},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2502.05947v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2502.05947.svg)](https://codeofpaper.com/paper/2502.05947)

Discussion