Shears: Unstructured Sparsity with Neural Low-rank Adapter Search

J. Pablo Muñoz, Jinjie Yuan, Nilesh Jain
4/16/2024
cs.LGcs.AIcs.CL

No official code found. Best implementation: IntelLabs/Hardware-Aware-Automated-Machine-Learning (76 stars, MIT, last commit 2026-04-13).

Abstract

Recently, several approaches successfully demonstrated that weight-sharing Neural Architecture Search (NAS) can effectively explore a search space of elastic low-rank adapters (LoRA), allowing the parameter-efficient fine-tuning (PEFT) and compression of large language models. In this paper, we introduce a novel approach called Shears, demonstrating how the integration of cost-effective sparsity and a proposed Neural Low-rank adapter Search (NLS) algorithm can further improve the efficiency of PEFT approaches. Results demonstrate the benefits of Shears compared to other methods, reaching high sparsity levels while improving or with little drop in accuracy, utilizing a single GPU for a pair of hours.

View on arXivView PDF

Code Implementations(1)

IntelLabs/Hardware-Aware-Automated-Machine-LearningCommunity100%
MIT
7616C++, ShellOct 10, 20226 months agoMIT

Similar Papers

Low-Rank Adapters Meet Neural Architecture Search for LLM Compression

88% match1 repo762025

Structural Pruning of Pre-trained Language Models via Neural Architecture Search

84% match1 repo82024

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

84% match2026

MoRe Fine-Tuning with 10x Fewer Parameters

84% match1 repo222024

Unlocking the Global Synergies in Low-Rank Adapters

83% match2024

Sparsity May Be All You Need: Sparse Random Parameter Adaptation

83% match1 repo2025

Cite this paper

@article{muoz2024shears,
  title  = {Shears: Unstructured Sparsity with Neural Low-rank Adapter Search},
  author = {J. Pablo Muñoz and Jinjie Yuan and Nilesh Jain},
  year   = {2024},
  eprint = {2404.10934},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.10934v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.10934.svg)](https://codeofpaper.com/paper/2404.10934)

Discussion