Trust the Model Where It Trusts Itself - Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption

Bernd Frauenknecht, Artur Eisele, Devdutt Subhasish, Friedrich Solowjow, Sebastian Trimpe
2/12/2026

No official code found. Best implementation: Data-Science-in-Mechanical-Engineering/macura (8 stars, MIT, last commit 2025-01-02).

Abstract

Dyna-style model-based reinforcement learning (MBRL) combines model-free agents with predictive transition models through model-based rollouts. This combination raises a critical question: 'When to trust your model?'; i.e., which rollout length results in the model providing useful data? Janner et al. (2019) address this question by gradually increasing rollout lengths throughout the training. While theoretically tempting, uniform model accuracy is a fallacy that collapses at the latest when extrapolating. Instead, we propose asking the question 'Where to trust your model?'. Using inherent model uncertainty to consider local accuracy, we obtain the Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption (MACURA) algorithm. We propose an easy-to-tune rollout mechanism and demonstrate substantial improvements in data efficiency and performance compared to state-of-the-art deep MBRL methods on the MuJoCo benchmark.

View on arXivView PDF

Code Implementations(1)

Data-Science-in-Mechanical-Engineering/macuraCommunity100%
MIT

Code for the paper: "Trust the Model Where It Trusts Itself - Model-Based Actor-Critic with Uncertainty Aware Rollout Adaption"

84PythonMay 28, 20241 years agoMIT

Similar Papers

On Rollouts in Model-Based Reinforcement Learning

85% match1 repo32025

Why long model-based rollouts are no reason for bad Q-value estimates

85% match2024

Plan To Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learning

84% match1 repo72023

Diminishing Return of Value Expansion Methods

82% match1 repo162024

Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption

82% match2026

Bounding-Box Inference for Error-Aware Model-Based Reinforcement Learning

81% match1 repo42024

Cite this paper

@article{frauenknecht2026trust,
  title  = {Trust the Model Where It Trusts Itself - Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption},
  author = {Bernd Frauenknecht and Artur Eisele and Devdutt Subhasish and Friedrich Solowjow and Sebastian Trimpe},
  year   = {2026},
  eprint = {2405.19014},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2405.19014},
  journal = {ICML 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.19014.svg)](https://codeofpaper.com/paper/2405.19014)

Discussion