PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers

Yibo Zhong, Haoxiang Jiang, Lincan Li, Ryumei Nakada, Tianci Liu +3 more
10/2/2024
cs.LGcs.CL

No official code found. Best implementation: yibozhong/peanut (11 stars, last commit 2025-12-25).

Abstract

Fine-tuning large pre-trained foundation models often yields excellent downstream performance but is prohibitively expensive when updating all parameters. Parameter-efficient fine-tuning (PEFT) methods such as LoRA alleviate this by introducing lightweight update modules, yet they commonly rely on weight-agnostic linear approximations, limiting their expressiveness. In this work, we propose PEANuT, a novel PEFT framework that introduces weight-aware neural tweakers, compact neural modules that generate task-adaptive updates conditioned on frozen pre-trained weights. PEANuT provides a flexible yet efficient way to capture complex update patterns without full model tuning. We theoretically show that PEANuT achieves equivalent or greater expressivity than existing linear PEFT methods with comparable or fewer parameters. Extensive experiments across four benchmarks with over twenty datasets demonstrate that PEANuT consistently outperforms strong baselines in both NLP and vision tasks, while maintaining low computational overhead.

View on arXivView PDF

Code Implementations(1)

yibozhong/peanutCommunity73%

(KDD 2026) PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers

111Shell, PythonSep 28, 202410 months ago

Similar Papers

NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning

83% match1 repo112025

From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devices

82% match2025

Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models

82% match1 repo102025

PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection

82% match2025

DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks

82% match2025

TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

82% match2025

Cite this paper

@article{zhong2024peanut,
  title  = {PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers},
  author = {Yibo Zhong and Haoxiang Jiang and Lincan Li and Ryumei Nakada and Tianci Liu and Linjun Zhang and Huaxiu Yao and Haoyu Wang},
  year   = {2024},
  eprint = {2410.01870},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.01870v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.01870.svg)](https://codeofpaper.com/paper/2410.01870)

Discussion