Hyperbolic Image-Text Representations

Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, Ramakrishna Vedantam
4/18/2023
cs.CVcs.LG

No official code found. Best implementation: facebookresearch/meru (203 stars, last commit 2023-08-23).

Abstract

Visual and linguistic concepts naturally organize themselves in a hierarchy, where a textual concept "dog" entails all images that contain dogs. Despite being intuitive, current large-scale vision and language models such as CLIP do not explicitly capture such hierarchy. We propose MERU, a contrastive model that yields hyperbolic representations of images and text. Hyperbolic spaces have suitable geometric properties to embed tree-like data, so MERU can better capture the underlying hierarchy in image-text datasets. Our results show that MERU learns a highly interpretable and structured representation space while being competitive with CLIP's performance on standard multi-modal tasks like image classification and image-text retrieval. Our code and models are available at https://www.github.com/facebookresearch/meru

View on arXivView PDF

Code Implementations(1)

facebookresearch/meruCommunity100%

Code for the paper "Hyperbolic Image-Text Representations", Desai et al, ICML 2023

20325PythonAug 18, 20233 years agoNOASSERTION

Similar Papers

Compositional Entailment Learning for Hyperbolic Vision-Language Models

88% match1 repo1092024

HMID-Net: An Exploration of Masked Image Modeling and Knowledge Distillation in Hyperbolic Space

84% match2025

Emergent Visual-Semantic Hierarchies in Image-Text Representations

83% match2 repos342026

Hyperbolic Contrastive Learning

83% match2023

ARGENT: Adaptive Hierarchical Image-Text Representations

82% match2026

Hyperbolic Learning with Multimodal Large Language Models

82% match2024

Cite this paper

@article{desai2023hyperbolic,
  title  = {Hyperbolic Image-Text Representations},
  author = {Karan Desai and Maximilian Nickel and Tanmay Rajpurohit and Justin Johnson and Ramakrishna Vedantam},
  year   = {2023},
  eprint = {2304.09172},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.09172v3},
  journal = {ICML 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.09172.svg)](https://codeofpaper.com/paper/2304.09172)

Discussion