Run LoRA Run: Faster and Lighter LoRA Implementations

Daria Cherniuk, Aleksandr Mikhalev, Ivan Oseledets
12/6/2023
cs.LG

No official code found. Best implementation: KamikaziZen/RunLoRA (13 stars, last commit 2024-11-21).

Abstract

LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers. This technique is used both for fine-tuning and full training of large language models. This paper presents the RunLoRA framework for efficient implementations of LoRA that significantly improves the speed of neural network training and fine-tuning using low-rank adapters. The proposed implementation optimizes the computation of LoRA operations based on dimensions of corresponding linear layer, layer input dimensions and lora rank by choosing best forward and backward computation graph based on FLOPs and time estimations, resulting in faster training without sacrificing accuracy. The experimental results show up to 28\% speedup on language modeling networks.

View on arXivView PDF

Code Implementations(1)

KamikaziZen/RunLoRACommunity74%

Faster and Lighter LoRA Implementations

131Python, Jupyter NotebookDec 8, 20231 years ago

Similar Papers

A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA

87% match2023

LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning

86% match2023

LoRA: Low-Rank Adaptation of Large Language Models

85% match1 repo13.7k2021

Low-Rank Adapters Meet Neural Architecture Search for LLM Compression

85% match1 repo762025

Efficient softmax approximation for GPUs

85% match1 repo3952016

Distillation-Supervised Convolutional Low-Rank Adaptation for Efficient Image Super-Resolution

85% match1 repo662025

Cite this paper

@article{cherniuk2023lora,
  title  = {Run LoRA Run: Faster and Lighter LoRA Implementations},
  author = {Daria Cherniuk and Aleksandr Mikhalev and Ivan Oseledets},
  year   = {2023},
  eprint = {2312.03415},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.03415v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.03415.svg)](https://codeofpaper.com/paper/2312.03415)

Discussion