InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models

Lichang Chen, Jiuhai Chen, T. Goldstein, Heng Huang, Tianyi Zhou
2/12/2026

No official code found. Best implementation: Lichang-Chen/InstructZero (201 stars, last commit 2024-07-23).

Abstract

Large language models~(LLMs) are instruction followers, but it can be challenging to find the best instruction for different situations, especially for black-box LLMs on which backpropagation is forbidden. Instead of directly optimizing the discrete instruction, we optimize a low-dimensional soft prompt applied to an open-source LLM to generate the instruction for the black-box LLM. On each iteration of the proposed method, which we call InstructZero, a soft prompt is converted into an instruction using the open-source LLM, which is then submitted to the black-box LLM for zero-shot evaluation, and the performance is sent to Bayesian optimization to produce new soft prompts improving the zero-shot performance. We evaluate InstructZero on different combinations of open-source LLMs and APIs including Vicuna and ChatGPT. Our results show that InstructZero outperforms SOTA auto-instruction methods across a variety of downstream tasks. Our code and data are publicly available at https://github.com/Lichang-Chen/InstructZero.

View on arXivView PDF

Code Implementations(1)

Lichang-Chen/InstructZeroCommunity100%

Official Implementation of InstructZero; the first framework to optimize bad prompts of ChatGPT(API LLMs) and finally obtain good prompts!

20116Shell, PythonMay 29, 20232 years ago
black-box-optimizationblack-box-tuningchatgptinstruction-optimizationlarge-language-model+1 more

Similar Papers

WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions

85% match1 repo9.5k2026

Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers

84% match1 repo532026

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

84% match1 repo9.5k2026

Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions

82% match1 repo212026

VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization

82% match1 repo92024

ACING: Actor-Critic for Instruction Learning in Black-Box LLMs

81% match1 repo72024

Cite this paper

@article{chen2026instructzero,
  title  = {InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models},
  author = {Lichang Chen and Jiuhai Chen and T. Goldstein and Heng Huang and Tianyi Zhou},
  year   = {2026},
  eprint = {2306.03082},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2306.03082},
  journal = {ICML 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.03082.svg)](https://codeofpaper.com/paper/2306.03082)

Discussion