VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval

Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, Yongping Xiong
6/6/2024
cs.IRcs.CLcs.CV

Official code: FlagOpen/FlagEmbedding (12,263 stars, MIT, last commit 2026-04-22). 1 more implementation below.

Abstract

Multi-modal retrieval becomes increasingly popular in practice. However, the existing retrievers are mostly text-oriented, which lack the capability to process visual information. Despite the presence of vision-language models like CLIP, the current methods are severely limited in representing the text-only and image-only data. In this work, we present a new embedding model VISTA for universal multi-modal retrieval. Our work brings forth threefold technical contributions. Firstly, we introduce a flexible architecture which extends a powerful text encoder with the image understanding capability by introducing visual token embeddings. Secondly, we develop two data generation strategies, which bring high-quality composed image-text to facilitate the training of the embedding model. Thirdly, we introduce a multi-stage training algorithm, which first aligns the visual token embedding with the text encoder using massive weakly labeled data, and then develops multi-modal representation capability using the generated composed image-text data. In our experiments, VISTA achieves superior performances across a variety of multi-modal retrieval tasks in both zero-shot and supervised settings. Our model, data, and source code are available at https://github.com/FlagOpen/FlagEmbedding.

View on arXivView PDF

Code Implementations(2)

FlagOpen/FlagEmbeddingOfficial100%
MIT

Retrieval and Retrieval-augmented LLMs

12,263939Shell, PythonAug 2, 20236 months agoMIT
embeddingsinformation-retrievalllmretrieval-augmented-generationsentence-embeddings+1 more
JUNJIE99/VISTA_Evaluation_FineTuningCommunity70%

Evaluation code and datasets for the ACL 2024 paper, VISTA: Visualized Text Embedding for Universal Multi-Modal Retrieval. The original code and model can be accessed at FlagEmbedding.

482Shell, PythonJul 8, 20241 years ago
hybrid-modal-retrievalmultimodal-retrievalvision-language-model

Similar Papers

ABC: Achieving Better Control of Multimodal Embeddings using VLMs

84% match1 repo202025

jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval

83% match2025

MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment

82% match2026

VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

82% match1 repo6822024

WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report

82% match1 repo2162026

E5-V: Universal Embeddings with Multimodal Large Language Models

82% match1 repo2752024

Cite this paper

@article{zhou2024vista,
  title  = {VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval},
  author = {Junjie Zhou and Zheng Liu and Shitao Xiao and Bo Zhao and Yongping Xiong},
  year   = {2024},
  eprint = {2406.04292},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.04292v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.04292.svg)](https://codeofpaper.com/paper/2406.04292)

Discussion