Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?

Bowen Zhao, Leo Parker Dirac, Paulina Varshavskaya
9/25/2024
cs.CVcs.CL

No official code found. Best implementation: groundlight/vlm-visual-demonstrations (2 stars, MIT, last commit 2024-09-27).

Abstract

Large vision-language models (VLMs) have become state-of-the-art for many computer vision tasks, with in-context learning (ICL) as a popular adaptation strategy for new ones. But can VLMs learn novel concepts purely from visual demonstrations, or are they limited to adapting to the output format of ICL examples? We propose a new benchmark we call Spatial Visual Ambiguity Tasks (SVAT) that challenges state-of-the-art VLMs to learn new visuospatial tasks in-context. We find that VLMs fail to do this zero-shot, and sometimes continue to fail after finetuning. However, adding simpler data to the training by curriculum learning leads to improved ICL performance.

View on arXivView PDF

Code Implementations(1)

groundlight/vlm-visual-demonstrationsCommunity100%
MIT
20Sep 16, 20242 years agoMIT

Similar Papers

The Spatial Blindspot of Vision-Language Models

89% match2026

Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models

86% match1 repo202025

SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models

86% match1 repo1012025

Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models

86% match2025

What's "up" with vision-language models? Investigating their struggle with spatial reasoning

86% match2 repos752023

Hidden in plain sight: VLMs overlook their visual representations

85% match2025

Cite this paper

@article{zhao2024vision,
  title  = {Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?},
  author = {Bowen Zhao and Leo Parker Dirac and Paulina Varshavskaya},
  year   = {2024},
  eprint = {2409.17080},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.17080v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.17080.svg)](https://codeofpaper.com/paper/2409.17080)

Discussion