Learning a model is paramount for sample efficiency in reinforcement learning control of PDEs

Stefan Werner, Sebastian Peitz
2/14/2023
cs.LG

No official code found. Best implementation: stweigand97/model-based-pde-control (4 stars, last commit 2024-07-31).

Abstract

The goal of this paper is to make a strong point for the usage of dynamical models when using reinforcement learning (RL) for feedback control of dynamical systems governed by partial differential equations (PDEs). To breach the gap between the immense promises we see in RL and the applicability in complex engineering systems, the main challenges are the massive requirements in terms of the training data, as well as the lack of performance guarantees. We present a solution for the first issue using a data-driven surrogate model in the form of a convolutional LSTM with actuation. We demonstrate that learning an actuated model in parallel to training the RL agent significantly reduces the total amount of required data sampled from the real system. Furthermore, we show that iteratively updating the model is of major importance to avoid biases in the RL training. Detailed ablation studies reveal the most important ingredients of the modeling process. We use the chaotic Kuramoto-Sivashinsky equation do demonstarte our findings.

View on arXivView PDF

Code Implementations(1)

stweigand97/model-based-pde-controlCommunity100%

Numerical Evidence for Sample Efficiency of Model-Based over Model-Free Reinforcement Learning Control of Partial Differential Equations [ECC'24]

40Shell, PythonMar 7, 20232 years ago
gymkuramoto-sivashinskymbpomodel-based-reinforcement-learningpartial-differential-equations+5 more

Similar Papers

Interpretable and Efficient Data-driven Discovery and Control of Distributed Systems

85% match1 repo2024

Data-assimilated model-informed reinforcement learning

85% match2025

Distributed Control of Partial Differential Equations Using Convolutional Reinforcement Learning

84% match1 repo142023

Data-Assimilated Model-Based Reinforcement Learning for Partially Observed Chaotic Flows

84% match2025

Model-based adaptation for sample efficient transfer in reinforcement learning control of parameter-varying systems

83% match2023

Reinforcement learning-based estimation for partial differential equations

83% match2023

Cite this paper

@article{werner2023learning,
  title  = {Learning a model is paramount for sample efficiency in reinforcement learning control of PDEs},
  author = {Stefan Werner and Sebastian Peitz},
  year   = {2023},
  eprint = {2302.07160},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.07160v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.07160.svg)](https://codeofpaper.com/paper/2302.07160)

Discussion