Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modeling

Yunqi Zhu, Xuebing Yang, Yuanyuan Wu, Wensheng Zhang
5/15/2023
cs.CLcs.AI

No official code found. Best implementation: zhuyunqi96/LoraLPrun (13 stars, last commit 2023-05-21).

Abstract

The increasing size of language models raises great research interests in parameter-efficient fine-tuning such as LoRA that freezes the pre-trained model, and injects small-scale trainable parameters for multiple downstream tasks (e.g., summarization, question answering and translation). To further enhance the efficiency of fine-tuning, we propose a framework that integrates LoRA and structured layer pruning. The integrated framework is validated on two created deidentified medical report summarization datasets based on MIMIC-IV-Note and two public medical dialogue datasets. By tuning 0.6% parameters of the original model and pruning over 30% Transformer-layers, our framework can reduce 50% of GPU memory usage and speed up 100% of the training phase, while preserving over 92% generation qualities on free-text sequence-to-sequence tasks.

View on arXivView PDF

Code Implementations(1)

zhuyunqi96/LoraLPrunCommunity100%
131PythonMay 13, 20233 years ago

Similar Papers

Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

86% match1 repo52023

Tailored-LLaMA: Optimizing Few-Shot Learning in Pruned LLaMA Models with Task-Specific Prompts

86% match2024

Parameter-Efficient Fine-Tuning for Medical Text Summarization: A Comparative Study of Lora, Prompt Tuning, and Full Fine-Tuning

85% match1 repo2026

Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir

85% match2026

Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models

85% match2023

Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models

85% match2024

Cite this paper

@article{zhu2023parameterefficient,
  title  = {Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modeling},
  author = {Yunqi Zhu and Xuebing Yang and Yuanyuan Wu and Wensheng Zhang},
  year   = {2023},
  eprint = {2305.08285},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.08285v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.08285.svg)](https://codeofpaper.com/paper/2305.08285)

Discussion