VSE++: Improving Visual-Semantic Embeddings with Hard Negatives

Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, Sanja Fidler
7/18/2017
cs.LGcs.CLcs.CV

Official code: fartashf/vsepp (523 stars, Apache-2.0, PyTorch, last commit 2021-12-08).

Abstract

We present a new technique for learning visual-semantic embeddings for cross-modal retrieval. Inspired by hard negative mining, the use of hard negatives in structured prediction, and ranking loss functions, we introduce a simple change to common loss functions used for multi-modal embeddings. That, combined with fine-tuning and use of augmented data, yields significant gains in retrieval performance. We showcase our approach, VSE++, on MS-COCO and Flickr30K datasets, using ablation studies and comparisons with existing methods. On MS-COCO our approach outperforms state-of-the-art methods by 8.8% in caption retrieval and 11.3% in image retrieval (at R@1).

View on arXivView PDF

Code Implementations(1)

fartashf/vseppOfficial100%
PyTorchApache-2.0

PyTorch Code for the paper "VSE++: Improving Visual-Semantic Embeddings with Hard Negatives"

523127PythonJun 29, 20174 years agoApache-2.0
bmvcnegativespaperpytorchvse

Similar Papers

UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning

85% match1 repo772025

COM3D: Leveraging Cross-View Correspondence and Cross-Modal Mining for 3D Retrieval

85% match2024

Dynamic Visual Semantic Sub-Embeddings and Fast Re-Ranking

85% match2023

Intra-Modal Constraint Loss For Image-Text Retrieval

84% match1 repo52022

Probabilistic Embeddings for Cross-Modal Retrieval

83% match1 repo1422021

Weakly supervised cross-modal learning in high-content screening

83% match1 repo102023

Cite this paper

@article{faghri2017vse,
  title  = {VSE++: Improving Visual-Semantic Embeddings with Hard Negatives},
  author = {Fartash Faghri and David J. Fleet and Jamie Ryan Kiros and Sanja Fidler},
  year   = {2017},
  eprint = {1707.05612},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1707.05612}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1707.05612.svg)](https://codeofpaper.com/paper/1707.05612)

Discussion