LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection

Zhuo Chen, Yufeng Huang, Jiaoyan Chen, Yuxia Geng, Yin Fang +3 more
7/26/2022
cs.CVcs.AI

No official code found. Best implementation: hackerchenzhuo/LaKo (24 stars, MIT, PyTorch, last commit 2024-02-09).

Abstract

Visual question answering (VQA) often requires an understanding of visual concepts and language semantics, which relies on external knowledge. Most existing methods exploit pre-trained language models or/and unstructured text, but the knowledge in these resources are often incomplete and noisy. Some other methods prefer to use knowledge graphs (KGs) which often have intensive structured knowledge, but the research is still quite preliminary. In this paper, we propose LaKo, a knowledge-driven VQA method via Late Knowledge-to-text Injection. To effectively incorporate an external KG, we transfer triples into textual format and propose a late injection mechanism for knowledge fusion. Finally we address VQA as a text generation task with an effective encoder-decoder paradigm, which achieves state-of-the-art results on OKVQA dataset.

View on arXivView PDF

Code Implementations(1)

hackerchenzhuo/LaKoCommunity100%
PyTorchMIT

[Paper][IJCKG 2022] LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection

243Shell, PythonNov 16, 20212 years agoMIT
encoder-decoderknowledgeknowledge-graphpytorchvisual+2 more

Similar Papers

Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering

87% match1 repo632023

Fusing Knowledge and Language: A Comparative Study of Knowledge Graph-Based Question Answering with LLMs

84% match2025

Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models

84% match2024

A Universal Question-Answering Platform for Knowledge Graphs

84% match1 repo152023

SelfGraphRAG: Bridging the Supervision Gap in Graph-Based RAG with Synthetic QA Generation

83% match2026

Massively Multilingual Language Models for Cross Lingual Fact Extraction from Low Resource Indian Languages

83% match2023

Cite this paper

@article{chen2022lako,
  title  = {LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection},
  author = {Zhuo Chen and Yufeng Huang and Jiaoyan Chen and Yuxia Geng and Yin Fang and Jeff Pan and Ningyu Zhang and Wen Zhang},
  year   = {2022},
  eprint = {2207.12888},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2207.12888v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2207.12888.svg)](https://codeofpaper.com/paper/2207.12888)

Discussion