CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts

Malvina Nikandrou, Georgios Pantazopoulos, Nikolas Vitsakis, Ioannis Konstas, Alessandro Suglia
10/20/2024
cs.CLcs.CV

No official code found. Best implementation: MalvinaNikandrou/crope (2 stars, MIT, last commit 2026-03-02).

Abstract

As Vision and Language models (VLMs) are reaching users across the globe, assessing their cultural understanding has become a critical challenge. In this paper, we introduce CROPE, a visual question answering benchmark designed to probe the knowledge of culture-specific concepts and evaluate the capacity for cultural adaptation through contextual information. This allows us to distinguish between parametric knowledge acquired during training and contextual knowledge provided during inference via visual and textual descriptions. Our evaluation of several state-of-the-art open VLMs shows large performance disparities between culture-specific and common concepts in the parametric setting. Moreover, experiments with contextual knowledge indicate that models struggle to effectively utilize multimodal information and bind culture-specific concepts to their depictions. Our findings reveal limitations in the cultural understanding and adaptability of current VLMs that need to be addressed toward more culturally inclusive models.

View on arXivView PDF

Code Implementations(1)

MalvinaNikandrou/cropeCommunity100%
MIT

Codebase for CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts

22Oct 20, 20247 months agoMIT

Similar Papers

Benchmarking Vision Language Models for Cultural Understanding

91% match2024

CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries

89% match2025

BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models

89% match1 repo42025

Seeing Culture: A Benchmark for Visual Reasoning and Grounding

87% match1 repo32025

Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration

87% match1 repo102024

From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models

87% match1 repo2024

Cite this paper

@article{nikandrou2024crope,
  title  = {CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts},
  author = {Malvina Nikandrou and Georgios Pantazopoulos and Nikolas Vitsakis and Ioannis Konstas and Alessandro Suglia},
  year   = {2024},
  eprint = {2410.15453},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.15453v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.15453.svg)](https://codeofpaper.com/paper/2410.15453)

Discussion