LoQT: Low-Rank Adapters for Quantized Pretraining

Sebastian Loeschcke, M. Toftrup, M. Kastoryano, Serge J. Belongie, Vésteinn Snæbjarnarson
2/3/2026

No official code found. Best implementation: sebulo/LoQT (82 stars, MIT, last commit 2024-11-11).

Abstract

Despite advances using low-rank adapters and quantization, pretraining of large models on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations, we propose Low-Rank Adapters for Quantized Training (LoQT), a method for efficiently training quantized models. LoQT uses gradient-based tensor factorization to initialize low-rank trainable weight matrices that are periodically merged into quantized full-rank weight matrices. Our approach is suitable for both pretraining and fine-tuning models. We demonstrate this for language modeling and downstream task adaptation, finding that LoQT enables efficient training of models up to 7B parameters on a 24GB GPU. We also demonstrate the feasibility of training a 13B model using per-layer gradient updates on the same hardware.

View on arXivView PDF

Code Implementations(1)

sebulo/LoQTCommunity100%
MIT
823Shell, PythonMay 26, 20241 years agoMIT

Similar Papers

LoQT: Low-Rank Adapters for Quantized Pretraining

100% match1 repo822024

Low-Rank Quantization-Aware Training for LLMs

88% match1 repo542024

LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning

87% match1 repo1292026

LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning

87% match1 repo1292023

LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices

85% match2024

L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models

85% match2024

Cite this paper

@article{loeschcke2026loqt,
  title  = {LoQT: Low-Rank Adapters for Quantized Pretraining},
  author = {Sebastian Loeschcke and M. Toftrup and M. Kastoryano and Serge J. Belongie and Vésteinn Snæbjarnarson},
  year   = {2026},
  eprint = {2405.16528},
  archivePrefix = {arXiv},
  doi    = {10.52202/079017-3661},
  url    = {https://arxiv.org/abs/2405.16528},
  journal = {NEURIPS 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.16528.svg)](https://codeofpaper.com/paper/2405.16528)

Discussion