Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson +2 more
7/25/2017
cs.CV

Official code: peteanderson80/bottom-up-attention (1,470 stars, MIT, last commit 2023-02-03).

Abstract

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

View on arXivView PDF

Code Implementations(1)

peteanderson80/bottom-up-attentionOfficial100%
MIT

Bottom-up attention model for image captioning and VQA, based on Faster R-CNN and Visual Genome

1,470370C, C++May 26, 20173 years agoMIT
caffecaptioning-imagesfaster-rcnnimage-captioningmscoco+3 more

Similar Papers

Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Models

85% match2025

Top-Down Visual Attention from Analysis by Synthesis

84% match1 repo1652026

Caption-Driven Explorations: Aligning Image and Text Embeddings through Human-Inspired Foveated Vision

82% match2024

Explainable Image Captioning using CNN- CNN architecture and Hierarchical Attention

82% match2024

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models

82% match1 repo142026

Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models

82% match2025

Cite this paper

@article{anderson2017bottomup,
  title  = {Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering},
  author = {Peter Anderson and Xiaodong He and Chris Buehler and Damien Teney and Mark Johnson and Stephen Gould and Lei Zhang},
  year   = {2017},
  eprint = {1707.07998},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1707.07998}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1707.07998.svg)](https://codeofpaper.com/paper/1707.07998)

Discussion