CLoVe: Encoding Compositional Language in Contrastive Vision-Language Models

Santiago Castro, Amir Ziai, Avneesh Saluja, Zhuoning Yuan, Rada Mihalcea
2/22/2024
cs.CVcs.CL

Official code: Netflix/clove (14 stars, last commit 2024-08-28).

Abstract

Recent years have witnessed a significant increase in the performance of Vision and Language tasks. Foundational Vision-Language Models (VLMs), such as CLIP, have been leveraged in multiple settings and demonstrated remarkable performance across several tasks. Such models excel at object-centric recognition yet learn text representations that seem invariant to word order, failing to compose known concepts in novel ways. However, no evidence exists that any VLM, including large-scale single-stream models such as GPT-4V, identifies compositions successfully. In this paper, we introduce a framework to significantly improve the ability of existing models to encode compositional language, with over 10% absolute improvement on compositionality benchmarks, while maintaining or improving the performance on standard object-recognition and retrieval benchmarks. Our code and pre-trained models are publicly available at https://github.com/netflix/clove.

View on arXivView PDF

Code Implementations(1)

Netflix/cloveOfficial100%
145Shell, PythonFeb 8, 20242 years agoNOASSERTION

Similar Papers

Weak-to-Strong Compositional Learning from Generative Models for Language-based Object Detection

86% match2024

Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models

86% match1 repo182026

Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition

86% match1 repo102024

Auto-Comp: An Automated Pipeline for Scalable Compositional Probing of Contrastive Vision-Language Models

84% match1 repo2026

SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality

84% match1 repo952023

Prompting Large Vision-Language Models for Compositional Reasoning

84% match2024

Cite this paper

@article{castro2024clove,
  title  = {CLoVe: Encoding Compositional Language in Contrastive Vision-Language Models},
  author = {Santiago Castro and Amir Ziai and Avneesh Saluja and Zhuoning Yuan and Rada Mihalcea},
  year   = {2024},
  eprint = {2402.15021},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.15021v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.15021.svg)](https://codeofpaper.com/paper/2402.15021)

Discussion