BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models

Yibin Wang, Haizhou Shi, Ligong Han, Dimitris N. Metaxas, Hao Wang
2/3/2026

No official code found. Best implementation: Wang-ML-Lab/bayesian-peft (40 stars, MIT, last commit 2026-02-04).

Abstract

Large Language Models (LLMs) often suffer from overconfidence during inference, particularly when adapted to downstream domain-specific tasks with limited data. Previous work addresses this issue by employing approximate Bayesian estimation after the LLMs are trained, enabling them to quantify uncertainty. However, such post-training approaches' performance is severely limited by the parameters learned during training. In this paper, we go beyond post-training Bayesianization and propose Bayesian Low-Rank Adaptation by Backpropagation (BLoB), an algorithm that continuously and jointly adjusts both the mean and covariance of LLM parameters throughout the whole fine-tuning process. Our empirical results verify the effectiveness of BLoB in terms of generalization and uncertainty estimation, when evaluated on both in-distribution and out-of-distribution data.

View on arXivView PDF

Code Implementations(1)

Wang-ML-Lab/bayesian-peftCommunity100%
MIT

Bayesian Low-Rank Adaptation of LLMs: BLoB [NeurIPS 2024] and TFB [NeurIPS 2025]

408Shell, PythonAug 17, 20248 months agoMIT
bayeisanbayesian-deep-learninglarge-language-modelsuncertainty-estimationuncertainty-quantification

Similar Papers

BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models

100% match2 repos402024

Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models

89% match1 repo102024

Training-Free Bayesianization for Low-Rank Adapters of Large Language Models

89% match1 repo402024

Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation

88% match1 repo32024

Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models

88% match2025

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

88% match2026

Cite this paper

@article{wang2026blob,
  title  = {BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models},
  author = {Yibin Wang and Haizhou Shi and Ligong Han and Dimitris N. Metaxas and Hao Wang},
  year   = {2026},
  eprint = {2406.11675},
  archivePrefix = {arXiv},
  doi    = {10.48550/arXiv.2406.11675},
  url    = {https://arxiv.org/abs/2406.11675},
  journal = {NEURIPS 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.11675.svg)](https://codeofpaper.com/paper/2406.11675)

Discussion