In-Context Learning Improves Compositional Understanding of Vision-Language Models

Matteo Nulli, Anesa Ibrahimi, Avik Pal, Hoshe Lee, Ivona Najdenkoska
7/22/2024
cs.CVcs.AI

No official code found. Best implementation: HoeZey/vlm-compositionality (4 stars, last commit 2024-09-11).

Abstract

Vision-Language Models (VLMs) have shown remarkable capabilities in a large number of downstream tasks. Nonetheless, compositional image understanding remains a rather difficult task due to the object bias present in training data. In this work, we investigate the reasons for such a lack of capability by performing an extensive bench-marking of compositional understanding in VLMs. We compare contrastive models with generative ones and analyze their differences in architecture, pre-training data, and training tasks and losses. Furthermore, we leverage In-Context Learning (ICL) as a way to improve the ability of VLMs to perform more complex reasoning and understanding given an image. Our extensive experiments demonstrate that our proposed approach outperforms baseline models across multiple compositional understanding datasets.

View on arXivView PDF

Code Implementations(1)

HoeZey/vlm-compositionalityCommunity100%
40Shell, PythonApr 12, 20242 years ago

Similar Papers

Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding

87% match1 repo562026

A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks

86% match2025

Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions

85% match1 repo72025

Weak-to-Strong Compositional Learning from Generative Models for Language-based Object Detection

85% match2024

MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models

85% match2024

Object-centric Binding in Contrastive Language-Image Pretraining

85% match2025

Cite this paper

@article{nulli2024incontext,
  title  = {In-Context Learning Improves Compositional Understanding of Vision-Language Models},
  author = {Matteo Nulli and Anesa Ibrahimi and Avik Pal and Hoshe Lee and Ivona Najdenkoska},
  year   = {2024},
  eprint = {2407.15487},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.15487v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.15487.svg)](https://codeofpaper.com/paper/2407.15487)

Discussion