SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers

Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Alexander Mikhalev, Alexander Panchenko +1 more
10/9/2024
cs.CLcs.AI

No official code found. Best implementation: sayankotor/sparse_grads (2 stars, last commit 2025-01-09).

Abstract

The performance of Transformer models has been enhanced by increasing the number of parameters and the length of the processed text. Consequently, fine-tuning the entire model becomes a memory-intensive process. High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and often overlook MLP blocks, which contain about half of the model parameters. We propose a new selective PEFT method, namely SparseGrad, that performs well on MLP blocks. We transfer layer gradients to a space where only about 1\% of the layer's elements remain significant. By converting gradients into a sparse structure, we reduce the number of updated parameters. We apply SparseGrad to fine-tune BERT and RoBERTa for the NLU task and LLaMa-2 for the Question-Answering task. In these experiments, with identical memory requirements, our method outperforms LoRA and MeProp, robust popular state-of-the-art PEFT approaches.

View on arXivView PDF

Code Implementations(1)

sayankotor/sparse_gradsCommunity100%

Sparse grads

20Python, Jupyter NotebookJul 14, 20231 years ago

Similar Papers

Sparse Gradient Compression for Fine-Tuning Large Language Models

84% match2025

Sparse Matrix in Large Language Model Fine-tuning

84% match2024

Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models

83% match1 repo92024

Sparsity May Be All You Need: Sparse Random Parameter Adaptation

83% match1 repo2025

Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models

83% match1 repo112024

Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models

83% match1 repo142024

Cite this paper

@article{chekalina2024sparsegrad,
  title  = {SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers},
  author = {Viktoriia Chekalina and Anna Rudenko and Gleb Mezentsev and Alexander Mikhalev and Alexander Panchenko and Ivan Oseledets},
  year   = {2024},
  eprint = {2410.07383},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.07383v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.07383.svg)](https://codeofpaper.com/paper/2410.07383)

Discussion