Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models

Emre Onal, Klemens Flöge, Emma Caldwell, Arsen Sheverdin, Vincent Fortuin
5/6/2024
cs.CL

No official code found. Best implementation: fortuinlab/swag-lora (10 stars, MIT, last commit 2025-08-11).

Abstract

Fine-tuned Large Language Models (LLMs) often suffer from overconfidence and poor calibration, particularly when fine-tuned on small datasets. To address these challenges, we propose a simple combination of Low-Rank Adaptation (LoRA) with Gaussian Stochastic Weight Averaging (SWAG), facilitating approximate Bayesian inference in LLMs. Through extensive testing across several Natural Language Processing (NLP) benchmarks, we demonstrate that our straightforward and computationally efficient approach improves model generalization and calibration competitively with comparable, more sophisticated methods for Bayesian inference in LLMs. We further show that our method exhibits greater robustness against distribution shift, as reflected in its improved performance on out-of-distribution tasks.

View on arXivView PDF

Code Implementations(1)

fortuinlab/swag-loraCommunity100%
MIT

Code for the paper "Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models".

102Shell, PythonApr 12, 20241 years agoMIT

Similar Papers

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

90% match2026

Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inference

89% match1 repo72025

BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models

89% match2 repos402024

BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models

89% match1 repo402026

Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models

88% match2025

Training-Free Bayesianization for Low-Rank Adapters of Large Language Models

87% match1 repo402024

Cite this paper

@article{onal2024gaussian,
  title  = {Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models},
  author = {Emre Onal and Klemens Flöge and Emma Caldwell and Arsen Sheverdin and Vincent Fortuin},
  year   = {2024},
  eprint = {2405.03425},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.03425v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.03425.svg)](https://codeofpaper.com/paper/2405.03425)

Discussion