R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning

Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu +5 more
5/22/2025
cs.CLcs.AIcs.IR

Official code: RUCAIBox/R1-Searcher (723 stars, MIT, last commit 2025-08-05). 1 more implementation below.

Abstract

Large Language Models (LLMs) are powerful but prone to hallucinations due to static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting external information, but current methods often are costly, generalize poorly, or ignore the internal knowledge of the model. In this paper, we introduce R1-Searcher++, a novel framework designed to train LLMs to adaptively leverage both internal and external knowledge sources. R1-Searcher++ employs a two-stage training strategy: an initial SFT Cold-start phase for preliminary format learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses outcome-supervision to encourage exploration, incorporates a reward mechanism for internal knowledge utilization, and integrates a memorization mechanism to continuously assimilate retrieved information, thereby enriching the model's internal knowledge. By leveraging internal knowledge and external search engine, the model continuously improves its capabilities, enabling efficient retrieval-augmented reasoning. Our experiments demonstrate that R1-Searcher++ outperforms previous RAG and reasoning methods and achieves efficient retrieval. The code is available at https://github.com/RUCAIBox/R1-Searcher-plus.

View on arXivView PDF

Code Implementations(2)

RUCAIBox/R1-SearcherOfficial100%
MIT

R1-searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning

72345Shell, PythonFeb 26, 20251 years agoMIT
RUCAIBox/R1-Searcher-plusOfficial100%
MIT

R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning

822PythonMay 22, 20251 years agoMIT

Similar Papers

R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning

95% match1 repo7232025

Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

93% match1 repo5.2k2025

R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning

90% match1 repo352025

MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval

89% match2025

Search-R3: Unifying Reasoning and Embedding Generation in Large Language Models

88% match1 repo142025

ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning

88% match1 repo1.4k2025

Cite this paper

@article{song2025rsearcher,
  title  = {R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning},
  author = {Huatong Song and Jinhao Jiang and Wenqing Tian and Zhipeng Chen and Yuhuan Wu and Jiahao Zhao and Yingqian Min and Wayne Xin Zhao and Lei Fang and Ji-Rong Wen},
  year   = {2025},
  eprint = {2505.17005},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.17005v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.17005.svg)](https://codeofpaper.com/paper/2505.17005)

Discussion