VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization

Dongsheng Zhu, Xunzhu Tang, Weidong Han, Jinghui Lu, Yukun Zhao +3 more
2/12/2024
cs.AI

Official code: Zhudongsheng75/VisLingInstruct (9 stars, last commit 2024-05-26).

Abstract

This paper presents VisLingInstruct, a novel approach to advancing Multi-Modal Language Models (MMLMs) in zero-shot learning. Current MMLMs show impressive zero-shot abilities in multi-modal tasks, but their performance depends heavily on the quality of instructions. VisLingInstruct tackles this by autonomously evaluating and optimizing instructional texts through In-Context Learning, improving the synergy between visual perception and linguistic expression in MMLMs. Alongside this instructional advancement, we have also optimized the visual feature extraction modules in MMLMs, further augmenting their responsiveness to textual content. Our comprehensive experiments on MMLMs, based on FlanT5 and Vicuna, show that VisLingInstruct significantly improves zero-shot performance in visual multi-modal tasks. Notably, it achieves a 13.1% and 9% increase in accuracy over the prior state-of-the-art on the TextVQA and HatefulMemes datasets. Our main code is available at https://github.com/Zhudongsheng75/VisLingInstruct.

View on arXivView PDF

Code Implementations(1)

Zhudongsheng75/VisLingInstructOfficial100%

(NAACL 2024) VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization

90Shell, PythonFeb 11, 20242 years ago

Similar Papers

Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning

88% match1 repo32026

Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models

85% match1 repo242026

Visual Programming for Zero-Shot Open-Vocabulary 3D Visual Grounding

85% match1 repo642026

DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training

85% match1 repo1452026

Towards Multimodal In-Context Learning for Vision & Language Models

84% match2024

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

84% match1 repo3532026

Cite this paper

@article{zhu2024vislinginstruct,
  title  = {VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization},
  author = {Dongsheng Zhu and Xunzhu Tang and Weidong Han and Jinghui Lu and Yukun Zhao and Guoliang Xing and Junfeng Wang and Dawei Yin},
  year   = {2024},
  eprint = {2402.07398},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.07398v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.07398.svg)](https://codeofpaper.com/paper/2402.07398)

Discussion