Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering

Alireza Salemi, Mahta Rafiee, Hamed Zamani
6/28/2023
cs.IRcs.CLcs.CV

No official code found. Best implementation: alirezasalemi7/pretraining-multimodal-dense-retriever-for-okvqa (5 stars, last commit 2023-08-22).

Abstract

This paper studies a category of visual question answering tasks, in which accessing external knowledge is necessary for answering the questions. This category is called outside-knowledge visual question answering (OK-VQA). A major step in developing OK-VQA systems is to retrieve relevant documents for the given multi-modal query. Current state-of-the-art asymmetric dense retrieval model for this task uses an architecture with a multi-modal query encoder and a uni-modal document encoder. Such an architecture requires a large amount of training data for effective performance. We propose an automatic data generation pipeline for pre-training passage retrieval models for OK-VQA tasks. The proposed approach leads to 26.9% Precision@5 improvements compared to the current state-of-the-art asymmetric architecture. Additionally, the proposed pre-training approach exhibits a good ability in zero-shot retrieval scenarios.

View on arXivView PDF

Code Implementations(1)

alirezasalemi7/pretraining-multimodal-dense-retriever-for-okvqaCommunity100%
50Jun 16, 20233 years ago

Similar Papers

Multimodal Iterative RAG for Knowledge-Intensive Visual Question Answering

85% match2025

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

85% match1 repo272023

BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answering

84% match2026

OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answering in Vietnamese

84% match1 repo62023

Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering

84% match1 repo222025

Cross-modal Retrieval for Knowledge-based Visual Question Answering

84% match1 repo392024

Cite this paper

@article{salemi2023pretraining,
  title  = {Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering},
  author = {Alireza Salemi and Mahta Rafiee and Hamed Zamani},
  year   = {2023},
  eprint = {2306.16478},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.16478v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.16478.svg)](https://codeofpaper.com/paper/2306.16478)

Discussion