Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis

Jia Lin Hau, Erick Delage, Esther Derman, Mohammad Ghavamzadeh, Marek Petrik
10/31/2024
cs.LG

No official code found. Best implementation: MonkieDein/DRA-Q-LA (0 stars, MIT, last commit 2026-03-19).

Abstract

In Markov decision processes (MDPs), quantile risk measures such as Value-at-Risk are a standard metric for modeling RL agents' preferences for certain outcomes. This paper proposes a new Q-learning algorithm for quantile optimization in MDPs with strong convergence and performance guarantees. The algorithm leverages a new, simple dynamic program (DP) decomposition for quantile MDPs. Compared with prior work, our DP decomposition requires neither known transition probabilities nor solving complex saddle point equations and serves as a suitable foundation for other model-free RL algorithms. Our numerical results in tabular domains show that our Q-learning algorithm converges to its DP variant and outperforms earlier algorithms.

View on arXivView PDF

Code Implementations(1)

MonkieDein/DRA-Q-LACommunity100%
MIT

Distributional Risk-Averse Quantile Q Learning

00Julia, ShellOct 1, 20247 months agoMIT

Similar Papers

Planning and Learning in Average Risk-aware MDPs

85% match2025

On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes

83% match2024

Optimistic Reinforcement Learning with Quantile Objectives

83% match2025

Quantizer Design for Finite Model Approximations, Model Learning, and Quantized Q-Learning for MDPs with Unbounded Spaces

82% match2025

Variance-Reduced Cascade Q-learning: Algorithms and Sample Complexity

82% match2024

Risk-Averse Total-Reward Reinforcement Learning

82% match1 repo2025

Cite this paper

@article{hau2024qlearning,
  title  = {Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis},
  author = {Jia Lin Hau and Erick Delage and Esther Derman and Mohammad Ghavamzadeh and Marek Petrik},
  year   = {2024},
  eprint = {2410.24128},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.24128v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.24128.svg)](https://codeofpaper.com/paper/2410.24128)

Discussion