Image Captioning: Transforming Objects into Words

Simao Herdade, Armin Kappeler, Kofi Boakye, Joao Soares
6/14/2019
cs.CVcs.CL

Official code: yahoo/object_relation_transformer (180 stars, MIT, last commit 2024-09-17).

Abstract

Image captioning models typically follow an encoder-decoder architecture which uses abstract image feature vectors as input to the encoder. One of the most successful algorithms uses feature vectors extracted from the region proposals obtained from an object detector. In this work we introduce the Object Relation Transformer, that builds upon this approach by explicitly incorporating information about the spatial relationship between input detected objects through geometric attention. Quantitative and qualitative results demonstrate the importance of such geometric attention for image captioning, leading to improvements on all common captioning metrics on the MS-COCO dataset.

View on arXivView PDF

Code Implementations(1)

yahoo/object_relation_transformerOfficial100%
MIT

Implementation of the Object Relation Transformer for Image Captioning

18046HTML, ShellOct 23, 20192 years agoMIT
machine-learning

Similar Papers

Attention on Attention for Image Captioning

84% match1 repo3402019

GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features

84% match1 repo1992022

DEVICE: Depth and Visual Concepts Aware Transformer for OCR-based Image Captioning

84% match2023

SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning

84% match1 repo2122016

Retrieval-Augmented Transformer for Image Captioning

83% match2022

SuperCap: Multi-resolution Superpixel-based Image Captioning

83% match2025

Cite this paper

@article{herdade2019image,
  title  = {Image Captioning: Transforming Objects into Words},
  author = {Simao Herdade and Armin Kappeler and Kofi Boakye and Joao Soares},
  year   = {2019},
  eprint = {1906.05963},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1906.05963}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1906.05963.svg)](https://codeofpaper.com/paper/1906.05963)

Discussion