Integrating Query-aware Segmentation and Cross-Attention for Robust VQA

Wonjun Choi, Sangbeom Lee, Seungyeon Lee, Heechul Jung, Dong-Gyu Lee
7/9/2024
cs.CV

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

This paper introduces a method for VizWiz-VQA using LVLM with trainable cross-attention and LoRA finetuning. We train the model with the following conditions: 1) Training with original images. 2) Training with enhanced images using CLIPSeg to highlight or contrast the original image. 3) Training with integrating the output features of Vision Transformer (ViT) and CLIPSeg features of the original images. Then, we ensemble the results based on Levenshtein distance to enhance the prediction of the final answer. In the experiments, we demonstrate and analyze the proposed method's effectiveness.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis

84% match1 repo602026

Prediction-Feedback DETR for Temporal Action Detection

83% match2024

EDIT: Enhancing Vision Transformers by Mitigating Attention Sink through an Encoder-Decoder Architecture

83% match2025

HAAT: Hybrid Attention Aggregation Transformer for Image Super-Resolution

83% match1 repo5452024

Learning Visual Prompts for Guiding the Attention of Vision Transformers

83% match2024

LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition

83% match1 repo102024

Cite this paper

@article{choi2024integrating,
  title  = {Integrating Query-aware Segmentation and Cross-Attention for Robust VQA},
  author = {Wonjun Choi and Sangbeom Lee and Seungyeon Lee and Heechul Jung and Dong-Gyu Lee},
  year   = {2024},
  eprint = {2407.12055},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.12055v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.12055.svg)](https://codeofpaper.com/paper/2407.12055)

Discussion