Token-Level Adaptation of LoRA Adapters for Downstream Task Generalization

Joshua Belofsky
11/17/2023
cs.CLcs.LG

Official code: jb-01/LoRA-TLE (15 stars, last commit 2024-04-14).

Abstract

This paper introduces a method for adapting LoRA adapters in smaller-sized language models to arbitrary downstream tasks. Unlike standard mixture-of-expert architectures, our method employs a gradient-free routing function to choose a weighted combination of experts without increasing the compute requirements for training or inference. The results show that token-level adaptation of LoRA adapters outperforms the base Llama-2-7b model across mathematical (GSM8K), scientific (ARC-Challenge), reading comprehension (SQuAD), and coding (CodeAlpaca-20k) tasks. Further evaluations also show that the average performance of token-level adaptation outperforms individual models fine-tuned for each of the tasks with the best performance observed in adaptation of every-other token during inference. The code for this study is made available through a public repository.

View on arXivView PDF

Code Implementations(1)

jb-01/LoRA-TLEOfficial100%

Token-level adaptation of LoRA matrices for downstream task generalization.

150PythonAug 20, 20232 years ago

Similar Papers

LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing

89% match2025

LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging

88% match2025

Ensembles of Low-Rank Expert Adapters

87% match2025

Text-to-LoRA: Instant Transformer Adaption

86% match1 repo1.3k2025

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

86% match2024

Layer-wise LoRA fine-tuning: a similarity metric approach

86% match1 repo42026

Cite this paper

@article{belofsky2023tokenlevel,
  title  = {Token-Level Adaptation of LoRA Adapters for Downstream Task Generalization},
  author = {Joshua Belofsky},
  year   = {2023},
  eprint = {2311.10847},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.10847v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.10847.svg)](https://codeofpaper.com/paper/2311.10847)

Discussion