RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data

Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Murun Yang +7 more
8/22/2024
cs.CVcs.CL

No official code found. Best implementation: wangclnlp/Vision-LLM-Alignment (3 stars, last commit 2025-06-18).

Abstract

Large vision-language models (LVLMs) often fail to align with human preferences, leading to issues like generating misleading content without proper visual context (also known as hallucination). A promising solution to this problem is using human-preference alignment techniques, such as best-of-n sampling and reinforcement learning. However, these techniques face the difficulty arising from the scarcity of visual preference data, which is required to train a visual reward model (VRM). In this work, we continue the line of research. We present a Robust Visual Reward Model (RoVRM) which improves human-preference alignment for LVLMs. RoVRM leverages auxiliary textual preference data through a three-phase progressive training and optimal transport-based preference data selection to effectively mitigate the scarcity of visual preference data. We experiment with RoVRM on the commonly used vision-language tasks based on the LLaVA-1.5-7B and -13B models. Experimental results demonstrate that RoVRM consistently outperforms traditional VRMs. Furthermore, our three-phase progressive training and preference data selection approaches can yield consistent performance gains over ranking-based alignment techniques, such as direct preference optimization.

View on arXivView PDF

Code Implementations(1)

wangclnlp/Vision-LLM-AlignmentCommunity100%

This repository contains the code for SFT, RLHF, and DPO, designed for vision-based LLMs, including the LLaVA models and the LLaMA-3.2-vision models.

30Shell, PythonSep 29, 20241 years ago

Similar Papers

Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization

90% match2025

Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization

89% match1 repo512025

Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards

89% match2025

Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning

88% match1 repo2492025

VLP: Vision-Language Preference Learning for Embodied Manipulation

87% match2025

From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models

87% match2025

Cite this paper

@article{wang2024rovrm,
  title  = {RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data},
  author = {Chenglong Wang and Yang Gan and Yifu Huo and Yongyu Mu and Murun Yang and Qiaozhi He and Tong Xiao and Chunliang Zhang and Tongran Liu and Quan Du and Di Yang and Jingbo Zhu},
  year   = {2024},
  eprint = {2408.12109},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.12109v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.12109.svg)](https://codeofpaper.com/paper/2408.12109)

Discussion