Small Models are Valuable Plug-ins for Large Language Models

Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu +1 more
5/15/2023
cs.CLcs.AIcs.LG

No official code found. Best implementation: JetRunner/SuperICL (131 stars, Apache-2.0, last commit 2023-05-16).

Abstract

Large language models (LLMs) such as GPT-3 and GPT-4 are powerful but their weights are often publicly unavailable and their immense sizes make the models difficult to be tuned with common hardware. As a result, effectively tuning these models with large-scale supervised data can be challenging. As an alternative, In-Context Learning (ICL) can only use a small number of supervised examples due to context length limits. In this paper, we propose Super In-Context Learning (SuperICL) which allows black-box LLMs to work with locally fine-tuned smaller models, resulting in superior performance on supervised tasks. Our experiments demonstrate that SuperICL can improve performance beyond state-of-the-art fine-tuned models while addressing the instability problem of in-context learning. Furthermore, SuperICL can enhance the capabilities of smaller models, such as multilinguality and interpretability.

View on arXivView PDF

Code Implementations(1)

JetRunner/SuperICLCommunity100%
Apache-2.0

Code for "Small Models are Valuable Plug-ins for Large Language Models"

1314PythonMay 15, 20233 years agoApache-2.0

Similar Papers

Improving In-Context Learning with Small Language Model Ensembles

91% match1 repo62024

Focused Large Language Models are Stable Many-Shot Learners

86% match2024

Emergent Abilities in Reduced-Scale Generative Language Models

86% match2024

Grimoire is All You Need for Enhancing Large Language Models

86% match1 repo1202024

Fine-Tuned In-Context Learners for Efficient Adaptation

86% match2025

You Only Fine-tune Once: Many-Shot In-Context Fine-Tuning for Large Language Model

86% match2025

Cite this paper

@article{xu2023small,
  title  = {Small Models are Valuable Plug-ins for Large Language Models},
  author = {Canwen Xu and Yichong Xu and Shuohang Wang and Yang Liu and Chenguang Zhu and Julian McAuley},
  year   = {2023},
  eprint = {2305.08848},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.08848v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.08848.svg)](https://codeofpaper.com/paper/2305.08848)

Discussion