LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models

Runming Yang, Taiqiang Wu, Jiahao Wang, Pengfei Hu, Yik-Chung Wu +2 more
11/11/2024
cs.CLcs.AIcs.LG

Official code: yang3121099/LLM-Neo (17 stars, Apache-2.0, last commit 2025-03-02).

Abstract

Knowledge distillation (KD) has been a predominant method for compressing Large Language Models (LLMs). In this paper, we first revisit KD and Low-Rank Adaption (LoRA) and demonstrate that they follow the same paradigm. Inspired by this observation, we propose a parameter-efficient knowledge distillation method, LLM-NEO, which integrates LoRA into KD to improve the efficiency of knowledge transfer. After that, we summarize some valuable guidelines for the hyperparameters in LLM-NEO. Experimental results on compressing Llama 2 and Llama 3.2 show that LLM-NEO outperforms various baselines. Further analysis demonstrates the robustness of the proposed LLM-NEO on variants of LoRA. The code and trained models are available at [Github](https://github.com/yang3121099/LLM-Neo).

View on arXivView PDF

Code Implementations(1)

yang3121099/LLM-NeoOfficial100%
Apache-2.0

The code for paper "LLM-Neo: Parameter Efficient Knowledge Distillation for Large Language Models"

174Shell, PythonFeb 24, 20251 years agoApache-2.0

Similar Papers

Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application

89% match2024

KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation

88% match1 repo242024

Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions

87% match2025

LLMR: Knowledge Distillation with a Large Language Model-Induced Reward

87% match2024

A Survey on Knowledge Distillation of Large Language Models

86% match1 repo1.3k2024

Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deployment

86% match2026

Cite this paper

@article{yang2024llmneo,
  title  = {LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models},
  author = {Runming Yang and Taiqiang Wu and Jiahao Wang and Pengfei Hu and Yik-Chung Wu and Ngai Wong and Yujiu Yang},
  year   = {2024},
  eprint = {2411.06839},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.06839v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.06839.svg)](https://codeofpaper.com/paper/2411.06839)

Discussion