VLRM: Vision-Language Models act as Reward Models for Image Captioning

Maksim Dzabraev, Alexander Kunitsyn, Andrei Ivaniuta
4/2/2024
cs.CV

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

In this work, we present an unsupervised method for enhancing an image captioning model (in our case, BLIP2) using reinforcement learning and vision-language models like CLIP and BLIP2-ITM as reward models. The RL-tuned model is able to generate longer and more comprehensive descriptions. Our model reaches impressive 0.90 R@1 CLIP Recall score on MS-COCO Carpathy Test Split. Weights are available at https://huggingface.co/sashakunitsyn/vlrm-blip2-opt-2.7b.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

ViTOC: Vision Transformer and Object-aware Captioner

88% match2024

Evcap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension

87% match1 repo632026

Captioning Daily Activity Images in Early Childhood Education: Benchmark and Algorithm

87% match2026

Fluent and Accurate Image Captioning with a Self-Trained Reward Model

87% match2024

Pretrained Image-Text Models are Secretly Video Captioners

86% match1 repo72025

Image-POSER: Reflective RL for Multi-Expert Image Generation and Editing

86% match2025

Cite this paper

@article{dzabraev2024vlrm,
  title  = {VLRM: Vision-Language Models act as Reward Models for Image Captioning},
  author = {Maksim Dzabraev and Alexander Kunitsyn and Andrei Ivaniuta},
  year   = {2024},
  eprint = {2404.01911},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.01911v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.01911.svg)](https://codeofpaper.com/paper/2404.01911)

Discussion