Image-text Retrieval via Preserving Main Semantics of Vision

Xu Zhang, Xinzheng Niu, Philippe Fournier-Viger, Xudong Dai
4/20/2023
cs.CV

Official code: ZhangXu0963/VSL (15 stars, last commit 2023-12-25).

Abstract

Image-text retrieval is one of the major tasks of cross-modal retrieval. Several approaches for this task map images and texts into a common space to create correspondences between the two modalities. However, due to the content (semantics) richness of an image, redundant secondary information in an image may cause false matches. To address this issue, this paper presents a semantic optimization approach, implemented as a Visual Semantic Loss (VSL), to assist the model in focusing on an image's main content. This approach is inspired by how people typically annotate the content of an image by describing its main content. Thus, we leverage the annotated texts corresponding to an image to assist the model in capturing the main content of the image, reducing the negative impact of secondary content. Extensive experiments on two benchmark datasets (MSCOCO and Flickr30K) demonstrate the superior performance of our method. The code is available at: https://github.com/ZhangXu0963/VSL.

View on arXivView PDF

Code Implementations(1)

ZhangXu0963/VSLOfficial100%

The code of "Image-text Retrieval via Preserving Main Semantic of Vision" in ICME 2023.

150PythonDec 5, 20222 years ago

Similar Papers

Rethinking Benchmarks for Cross-modal Image-text Retrieval

87% match1 repo282023

Scene Graph Based Fusion Network For Image-Text Retrieval

86% match2023

AToMiC: An Image/Text Retrieval Test Collection to Support Multimedia Content Creation

84% match2 repos1.1k2023

Unifying Two-Stream Encoders with Transformers for Cross-Modal Retrieval

84% match1 repo272023

TSVC:Tripartite Learning with Semantic Variation Consistency for Robust Image-Text Retrieval

84% match2025

ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval

84% match1 repo282022

Cite this paper

@article{zhang2023imagetext,
  title  = {Image-text Retrieval via Preserving Main Semantics of Vision},
  author = {Xu Zhang and Xinzheng Niu and Philippe Fournier-Viger and Xudong Dai},
  year   = {2023},
  eprint = {2304.10254},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.10254v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.10254.svg)](https://codeofpaper.com/paper/2304.10254)

Discussion