Do Vision and Language Models Share Concepts? A Vector Space Alignment Study

Jiaang Li, Yova Kementchedjhieva, Constanza Fierro, Anders Søgaard
2/13/2023
cs.CLcs.AIcs.CV

No official code found. Best implementation: jiaangli/VLCA (16 stars, Apache-2.0, last commit 2024-11-22).

Abstract

Large-scale pretrained language models (LMs) are said to ``lack the ability to connect utterances to the world'' (Bender and Koller, 2020), because they do not have ``mental models of the world' '(Mitchell and Krakauer, 2023). If so, one would expect LM representations to be unrelated to representations induced by vision models. We present an empirical evaluation across four families of LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge towards representations isomorphic to those of vision models, subject to dispersion, polysemy and frequency. This has important implications for both multi-modal processing and the LM understanding debate (Mitchell and Krakauer, 2023).

View on arXivView PDF

Code Implementations(1)

jiaangli/VLCACommunity100%
Apache-2.0

[TACL/EMNLP'24] Do Vision and Language Models Share Concepts? A Vector Space Alignment Study

160PythonSep 7, 20231 years agoApache-2.0

Similar Papers

How Visual Representations Map to Language Feature Space in Multimodal LLMs

87% match2025

Intriguing Properties of Large Language and Vision Models

87% match1 repo82024

Can We Talk Models Into Seeing the World Differently?

86% match1 repo302024

Representations in vision and language converge in a shared, multidimensional space of perceived similarities

86% match2025

Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability

86% match2025

Attention Guided Alignment in Efficient Vision-Language Models

86% match2025

Cite this paper

@article{li2023vision,
  title  = {Do Vision and Language Models Share Concepts? A Vector Space Alignment Study},
  author = {Jiaang Li and Yova Kementchedjhieva and Constanza Fierro and Anders Søgaard},
  year   = {2023},
  eprint = {2302.06555},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.06555v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.06555.svg)](https://codeofpaper.com/paper/2302.06555)

Discussion