Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding

Sangmin Bae, Jongwoo Ko, Hwanjun Song, Se-Young Yun
10/9/2023
cs.CL

No official code found. Best implementation: raymin0223/fast_robust_early_exit (67 stars, last commit 2024-09-28).

Abstract

To tackle the high inference latency exhibited by autoregressive language models, previous studies have proposed an early-exiting framework that allocates adaptive computation paths for each token based on the complexity of generating the subsequent token. However, we observed several shortcomings, including performance degradation caused by a state copying mechanism or numerous exit paths, and sensitivity to exit confidence thresholds. Consequently, we propose a Fast and Robust Early-Exiting (FREE) framework, which incorporates a shallow-deep module and a synchronized parallel decoding. Our framework enables faster inference by synchronizing the decoding process of the current token with previously stacked early-exited tokens. Furthermore, as parallel decoding allows us to observe predictions from both shallow and deep models, we present a novel adaptive threshold estimator that exploits a Beta mixture model to determine suitable confidence thresholds. We empirically demonstrated the superiority of our proposed framework on extensive generation tasks.

View on arXivView PDF

Code Implementations(1)

raymin0223/fast_robust_early_exitCommunity100%

Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)

6710Shell, PythonJun 26, 20232 years ago
autoregressive-modelsearly-exitingefficient-inferencellmsnlp

Similar Papers

Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling

89% match2026

R²-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction

88% match2026

CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference

88% match2026

FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference

88% match2026

SPEED: Speculative Pipelined Execution for Efficient Decoding

88% match2023

Accelerating Large Language Model Inference via Early-Exiting Algorithms

88% match2025

Cite this paper

@article{bae2023fast,
  title  = {Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding},
  author = {Sangmin Bae and Jongwoo Ko and Hwanjun Song and Se-Young Yun},
  year   = {2023},
  eprint = {2310.05424},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.05424v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.05424.svg)](https://codeofpaper.com/paper/2310.05424)

Discussion