LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents

Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li +8 more
11/9/2023
cs.CVcs.AIcs.CL

No official code found. Best implementation: LLaVA-VL/LLaVA-Plus-Codebase (769 stars, Apache-2.0, last commit 2024-02-01).

Abstract

LLaVA-Plus is a general-purpose multimodal assistant that expands the capabilities of large multimodal models. It maintains a skill repository of pre-trained vision and vision-language models and can activate relevant tools based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on multimodal instruction-following data to acquire the ability to use tools, covering visual understanding, generation, external knowledge retrieval, and compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in existing capabilities and exhibits new ones. It is distinct in that the image query is directly grounded and actively engaged throughout the entire human-AI interaction sessions, significantly improving tool use performance and enabling new scenarios.

View on arXivView PDF

Code Implementations(1)

LLaVA-VL/LLaVA-Plus-CodebaseCommunity100%
Apache-2.0

LLaVA-Plus: Large Language and Vision Assistants that Plug and Learn to Use Skills

76960CSS, HTMLNov 7, 20232 years agoApache-2.0
agentlarge-language-modelslarge-multimodal-modelsmultimodal-large-language-modelstool-use

Similar Papers

LLaVA-OneVision: Easy Visual Task Transfer

84% match1 repo4.5k2024

LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model

82% match1 repo4012024

LLaVAC: Fine-tuning LLaVA as a Multimodal Sentiment Classifier

82% match1 repo2025

Improved Baselines with Visual Instruction Tuning

82% match2023

LLaVA-Docent: Instruction Tuning with Multimodal Large Language Model to Support Art Appreciation Education

82% match2024

Visual Instruction Tuning

81% match1 repo24.9k2023

Cite this paper

@article{liu2023llavaplus,
  title  = {LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents},
  author = {Shilong Liu and Hao Cheng and Haotian Liu and Hao Zhang and Feng Li and Tianhe Ren and Xueyan Zou and Jianwei Yang and Hang Su and Jun Zhu and Lei Zhang and Jianfeng Gao and Chunyuan Li},
  year   = {2023},
  eprint = {2311.05437},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.05437v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.05437.svg)](https://codeofpaper.com/paper/2311.05437)

Discussion