The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning

Sheila Schoepp, Masoud Jafaripour, Yingyue Cao, Tianpei Yang, Fatemeh Abdollahi +4 more
2/21/2025
cs.LGcs.AIcs.CL

No official code found. Best implementation: Masoudjafaripour/FM_RL_Survey (8 stars, MIT, last commit 2025-06-07).

Abstract

Reinforcement learning (RL) has shown impressive results in sequential decision-making tasks. Meanwhile, Large Language Models (LLMs) and Vision-Language Models (VLMs) have emerged, exhibiting impressive capabilities in multimodal understanding and reasoning. These advances have led to a surge of research integrating LLMs and VLMs into RL. In this survey, we review representative works in which LLMs and VLMs are used to overcome key challenges in RL, such as lack of prior knowledge, long-horizon planning, and reward design. We present a taxonomy that categorizes these LLM/VLM-assisted RL approaches into three roles: agent, planner, and reward. We conclude by exploring open problems, including grounding, bias mitigation, improved representations, and action advice. By consolidating existing research and identifying future directions, this survey establishes a framework for integrating LLMs and VLMs into RL, advancing approaches that unify natural language and visual understanding with sequential decision-making.

View on arXivView PDF

Code Implementations(1)

Masoudjafaripour/FM_RL_SurveyCommunity67%
MIT

A repo for survey paper "The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning" and a collection of AWESOME papers focused on using LLMs, VLMs for improving RL.

80May 11, 20251 years agoMIT
llmllm-rlllmsreinforcement-learningrl+4 more

Similar Papers

Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning

89% match1 repo4172024

Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods

88% match2024

Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme

88% match1 repo1492025

World Models with Hints of Large Language Models for Goal Achieving

87% match1 repo22024

The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models

87% match2024

Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations

87% match2025

Cite this paper

@article{schoepp2025evolving,
  title  = {The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning},
  author = {Sheila Schoepp and Masoud Jafaripour and Yingyue Cao and Tianpei Yang and Fatemeh Abdollahi and Shadan Golestan and Zahin Sufiyan and Osmar R. Zaiane and Matthew E. Taylor},
  year   = {2025},
  eprint = {2502.15214},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2502.15214v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2502.15214.svg)](https://codeofpaper.com/paper/2502.15214)

Discussion