Sparse is Enough in Fine-tuning Pre-trained Large Language Model

Weixi Song, Z. Li, Lefei Zhang, Hai Zhao, Bo Du
2/12/2026

No official code found. Best implementation: song-wx/SIFT (24 stars, last commit 2024-06-26).

Abstract

With the prevalence of pre-training-fine-tuning paradigm, how to efficiently adapt the pre-trained model to the downstream tasks has been an intriguing issue. Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed for low-cost adaptation. Although PEFT has demonstrated effectiveness and been widely applied, the underlying principles are still unclear. In this paper, we adopt the PAC-Bayesian generalization error bound, viewing pre-training as a shift of prior distribution which leads to a tighter bound for generalization error. We validate this shift from the perspectives of oscillations in the loss landscape and the quasi-sparsity in gradient distribution. Based on this, we propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT), and validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning. The code is accessible at https://github.com/song-wx/SIFT/.

View on arXivView PDF

Code Implementations(1)

song-wx/SIFTCommunity100%

[ICML2024 Spotlight] Fine-Tuning Pre-trained Large Language Models Sparsely

242Shell, PythonOct 22, 20232 years ago

Similar Papers

PACE: marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization

89% match1 repo192026

SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation

87% match1 repo1202024

LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views

87% match2026

Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach

86% match1 repo272026

LARGO: Low-Rank Regulated Gradient Projection for Robust Parameter Efficient Fine-Tuning

85% match2025

FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning

85% match2026

Cite this paper

@article{song2026sparse,
  title  = {Sparse is Enough in Fine-tuning Pre-trained Large Language Model},
  author = {Weixi Song and Z. Li and Lefei Zhang and Hai Zhao and Bo Du},
  year   = {2026},
  eprint = {2312.11875},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2312.11875},
  journal = {ICML 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.11875.svg)](https://codeofpaper.com/paper/2312.11875)

Discussion