Nearest Neighbor Normalization Improves Multimodal Retrieval

Neil Chowdhury, Franklin Wang, Sumedh Shenoy, Douwe Kiela, Sarah Schwettmann +1 more
10/31/2024
cs.CVcs.AIcs.CL

No official code found. Best implementation: multimodal-interpretability/nnn (21 stars, MIT, last commit 2024-11-01).

Abstract

Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval. In this paper, we present a simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization (NNN). We show an improvement on retrieval metrics in both text retrieval and image retrieval for all of the contrastive models that we tested (CLIP, BLIP, ALBEF, SigLIP, BEiT) and for both of the datasets that we used (MS-COCO and Flickr30k). NNN requires a reference database, but does not require any training on this database, and can even increase the retrieval accuracy of a model after finetuning.

View on arXivView PDF

Code Implementations(1)

multimodal-interpretability/nnnCommunity100%
MIT

Nearest Neighbor Normalization (EMNLP 2024)

211PythonOct 1, 20241 years agoMIT

Similar Papers

Is Cross-modal Information Retrieval Possible without Training?

82% match2023

SCOT: Self-Supervised Contrastive Pretraining For Zero-Shot Compositional Retrieval

82% match1 repo2025

Optimizing CLIP Models for Image Retrieval with Maintained Joint-Embedding Alignment

82% match1 repo102024

MultiModal Fine-tuning with Synthetic Captions

81% match1 repo22026

CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning

81% match2 repos142024

Improving fine-grained understanding in image-text pre-training

81% match2024

Cite this paper

@article{chowdhury2024nearest,
  title  = {Nearest Neighbor Normalization Improves Multimodal Retrieval},
  author = {Neil Chowdhury and Franklin Wang and Sumedh Shenoy and Douwe Kiela and Sarah Schwettmann and Tristan Thrush},
  year   = {2024},
  eprint = {2410.24114},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.24114v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.24114.svg)](https://codeofpaper.com/paper/2410.24114)

Discussion