What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases

Anthony Meng Huat Tiong, Junqi Zhao, Boyang Li, Junnan Li, Steven C. H. Hoi +1 more
4/3/2024
cs.CV

Official code: jq-zh/olive-dataset (6 stars, MIT, last commit 2024-04-26).

Abstract

Vision-language (VL) models, pretrained on colossal image-text datasets, have attained broad VL competence that is difficult to evaluate. A common belief is that a small number of VL skills underlie the variety of VL tests. In this paper, we perform a large-scale transfer learning experiment aimed at discovering latent VL skills from data. We reveal interesting characteristics that have important implications for test suite design. First, generation tasks suffer from a length bias, suggesting benchmarks should balance tasks with varying output lengths. Second, we demonstrate that factor analysis successfully identifies reasonable yet surprising VL skill factors, suggesting benchmarks could leverage similar analyses for task selection. Finally, we present a new dataset, OLIVE (https://github.com/jq-zh/olive-dataset), which simulates user instructions in the wild and presents challenges dissimilar to all datasets we tested. Our findings contribute to the design of balanced and broad-coverage vision-language evaluation methods.

View on arXivView PDF

Code Implementations(1)

jq-zh/olive-datasetOfficial100%
MIT

The OLIVE dataset, containing 9,450 images, 30,120 unique instructions, and 47,250 responses, simulates in-the-wild user queries to vision-language models.

60Mar 21, 20242 years agoMIT

Similar Papers

Conditional Prototype Rectification Prompt Learning

88% match1 repo62024

Scaling Pre-training to One Hundred Billion Data for Vision Language Models

88% match2025

POINTS: Improving Your Vision-language Model with Affordable Strategies

87% match2024

PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models

86% match2025

Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs

86% match1 repo442025

Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models

85% match2026

Cite this paper

@article{tiong2024what,
  title  = {What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases},
  author = {Anthony Meng Huat Tiong and Junqi Zhao and Boyang Li and Junnan Li and Steven C. H. Hoi and Caiming Xiong},
  year   = {2024},
  eprint = {2404.02415},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.02415v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.02415.svg)](https://codeofpaper.com/paper/2404.02415)

Discussion