Bayesian low-rank adaptation for large language models

Adam X. Yang, Maxime Robeyns, Xi Wang, L. Aitchison
2/12/2026

No official code found. Best implementation: MaximeRobeyns/bayesian_lora (41 stars, Apache-2.0, last commit 2024-06-22). 1 more below.

Abstract

Low-rank adaptation (LoRA) has emerged as a new paradigm for cost-efficient fine-tuning of large language models (LLMs). However, fine-tuned LLMs often become overconfident especially when fine-tuned on small datasets. Bayesian methods, with their inherent ability to estimate uncertainty, serve as potent tools to mitigate overconfidence and enhance calibration. In this work, we introduce Laplace-LoRA, which applies a Bayesian approach to the LoRA parameters. Specifically, Laplace-LoRA applies a Laplace approximation to the posterior over the LoRA parameters, considerably improving the calibration of fine-tuned LLMs.

View on arXivView PDF

Code Implementations(2)

adamxyang/laplace-loraCommunity100%
MIT

Bayesian low-rank adaptation for large language models

2910Shell, PythonSep 29, 20232 years agoMIT
MaximeRobeyns/bayesian_loraCommunity65%
Apache-2.0

Bayesian Low-Rank Adaptation for Large Language Models

416Python, MakefileJan 19, 20242 years agoApache-2.0
kfaclaplace-approximationllm

Similar Papers

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

88% match2026

Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation

88% match1 repo32024

LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning

88% match1 repo92026

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

87% match2024

Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models

87% match1 repo102024

Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models

86% match2025

Cite this paper

@article{yang2026bayesian,
  title  = {Bayesian low-rank adaptation for large language models},
  author = {Adam X. Yang and Maxime Robeyns and Xi Wang and L. Aitchison},
  year   = {2026},
  eprint = {2308.13111},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/pdf/2308.13111},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.13111.svg)](https://codeofpaper.com/paper/2308.13111)

Discussion