SPICE: Semantic Propositional Image Caption Evaluation

Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould
7/29/2016
cs.CVcs.CL

Official code: peteanderson80/SPICE (148 stars, AGPL-3.0, last commit 2023-02-02).

Abstract

There is considerable interest in the task of automatically generating image captions. However, evaluation is challenging. Existing automatic evaluation metrics are primarily sensitive to n-gram overlap, which is neither necessary nor sufficient for the task of simulating human judgment. We hypothesize that semantic propositional content is an important component of human caption evaluation, and propose a new automated caption evaluation metric defined over scene graphs coined SPICE. Extensive evaluations across a range of models and datasets indicate that SPICE captures human judgments over model-generated captions better than other automatic metrics (e.g., system-level correlation of 0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and 0.53 for METEOR). Furthermore, SPICE can answer questions such as `which caption-generator best understands colors?' and `can caption-generators count?'

View on arXivView PDF

Code Implementations(1)

peteanderson80/SPICEOfficial100%
AGPL-3.0

Semantic Propositional Image Caption Evaluation

14831JavaJul 15, 20163 years agoAGPL-3.0
captioning-imagesimage-captioningmscoco

Similar Papers

JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models

92% match1 repo92023

CLAIR: Evaluating Image Captions with Large Language Models

90% match2023

A Novel Evaluation Framework for Image2Text Generation

89% match2024

Benchmarking and Improving Detail Image Caption

89% match1 repo862024

Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction

89% match2024

Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis

89% match2024

Cite this paper

@article{anderson2016spice,
  title  = {SPICE: Semantic Propositional Image Caption Evaluation},
  author = {Peter Anderson and Basura Fernando and Mark Johnson and Stephen Gould},
  year   = {2016},
  eprint = {1607.08822},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1607.08822}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1607.08822.svg)](https://codeofpaper.com/paper/1607.08822)

Discussion