An Evaluation of GPT-4V and Gemini in Online VQA

Mengchen Liu, Chongyan Chen, Danna Gurari
12/17/2023
cs.CVcs.AI

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

While there is much excitement about the potential of large multimodal models (LMM), a comprehensive evaluation is critical to establish their true capabilities and limitations. In support of this aim, we evaluate two state-of-the-art LMMs, GPT-4V and Gemini, on a new visual question answering dataset sourced from an authentic online question answering community. We conduct fine-grained analysis by generating seven types of metadata for nearly 2,000 visual questions, such as image type and the required image processing capabilities. Our zero-shot performance analysis highlights the types of questions that are most challenging for both models, including questions related to "puzzling" topic, with "Identification" user intention, with "Sheet Music" image type, or labeled as "hard" by GPT-4.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

87% match1 repo5652023

A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering

86% match1 repo172023

Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping

85% match1 repo112024

BinaryVQA: A Versatile Test Set to Evaluate the Out-of-Distribution Generalization of VQA Models

85% match2023

SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation

85% match1 repo132024

HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language

85% match2023

Cite this paper

@article{liu2023evaluation,
  title  = {An Evaluation of GPT-4V and Gemini in Online VQA},
  author = {Mengchen Liu and Chongyan Chen and Danna Gurari},
  year   = {2023},
  eprint = {2312.10637},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.10637v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.10637.svg)](https://codeofpaper.com/paper/2312.10637)

Discussion