EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models

Xuchen Pan, Yanxi Chen, Yaliang Li, Bolin Ding, Jingren Zhou
2/1/2024
cs.LGcs.AIcs.CL

Official code: pan-x-c/EE-LLM (82 stars, last commit 2024-06-14).

Abstract

This work introduces EE-Tuning, a lightweight and economical solution to training/tuning early-exit large language models (LLMs). In contrast to the common approach of full-parameter pre-training, EE-Tuning augments any pre-trained (and possibly fine-tuned) standard LLM with additional early-exit layers that are tuned in a parameter-efficient manner, which requires significantly less computational resources and training data. Our implementation of EE-Tuning achieves outstanding training efficiency via extensive performance optimizations, as well as scalability due to its full compatibility with 3D parallelism. Results of systematic experiments validate the efficacy of EE-Tuning, confirming that effective early-exit LLM inference can be achieved with a limited training budget. In hope of making early-exit LLMs accessible to the community, we release the source code of our implementation of EE-Tuning at https://github.com/pan-x-c/EE-LLM.

View on arXivView PDF

Code Implementations(1)

pan-x-c/EE-LLMOfficial100%

EE-LLM is a framework for large-scale training and inference of early-exit (EE) large language models (LLMs).

827C, C++Dec 7, 20232 years agoNOASSERTION

Similar Papers

Early Exit Is a Natural Capability in Transformer-based Models: An Empirical Study on Early Exit without Joint Optimization

86% match2024

STEP: Staged Parameter-Efficient Pre-training for Large Language Models

85% match2025

Accelerating Large Language Model Inference with Self-Supervised Early Exits

85% match2024

Benchmarking Optimizers for Large Language Model Pretraining

84% match2025

Large Language Models for Tuning Evolution Strategies

84% match2024

Revealing the Power of Post-Training for Small Language Models via Knowledge Distillation

83% match2025

Cite this paper

@article{pan2024eetuning,
  title  = {EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models},
  author = {Xuchen Pan and Yanxi Chen and Yaliang Li and Bolin Ding and Jingren Zhou},
  year   = {2024},
  eprint = {2402.00518},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.00518v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.00518.svg)](https://codeofpaper.com/paper/2402.00518)

Discussion