Distributionally Robust Deep Q-Learning

Chung I Lu, Julian Sester, Aijia Zhang
5/25/2025
cs.LGmath.OCq-fin.PM

No official code found. Best implementation: luchungi/Sinkhorn_RDQN (4 stars, last commit 2025-09-25).

Abstract

We propose a novel distributionally robust $Q$-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterized with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S\&{P}~500 data.

View on arXivView PDF

Code Implementations(1)

luchungi/Sinkhorn_RDQNCommunity100%

Robust DQN using the Sinkhorn distance

43Python, Jupyter NotebookMay 21, 20251 years ago

Similar Papers

Q-Learning with Shift-Aware Upper Confidence Bound in Non-Stationary Reinforcement Learning

85% match2025

Sample Complexity of Variance-reduced Distributionally Robust Q-learning

85% match2023

Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model

83% match1 repo2024

Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction

82% match2025

Deep neural networks can provably solve Bellman equations for Markov decision processes without the curse of dimensionality

82% match2025

Deep Transfer Q-Learning for Offline Non-Stationary Reinforcement Learning

82% match2025

Cite this paper

@article{lu2025distributionally,
  title  = {Distributionally Robust Deep Q-Learning},
  author = {Chung I Lu and Julian Sester and Aijia Zhang},
  year   = {2025},
  eprint = {2505.19058},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.19058v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.19058.svg)](https://codeofpaper.com/paper/2505.19058)

Discussion