Risk-Averse Total-Reward Reinforcement Learning

Xihong Su, Jia Lin Hau, Gersi Doko, Kishan Panaganti, Marek Petrik
6/26/2025
cs.LG

No official code found. Best implementation: suxh2019/ERM_EVaR_Q (0 stars, MIT, last commit 2026-03-23).

Abstract

Risk-averse total-reward Markov Decision Processes (MDPs) offer a promising framework for modeling and solving undiscounted infinite-horizon objectives. Existing model-based algorithms for risk measures like the entropic risk measure (ERM) and entropic value-at-risk (EVaR) are effective in small problems, but require full access to transition probabilities. We propose a Q-learning algorithm to compute the optimal stationary policy for total-reward ERM and EVaR objectives with strong convergence and performance guarantees. The algorithm and its optimality are made possible by ERM's dynamic consistency and elicitability. Our numerical results on tabular domains demonstrate quick and reliable convergence of the proposed Q-learning algorithm to the optimal risk-averse value function.

View on arXivView PDF

Code Implementations(1)

suxh2019/ERM_EVaR_QCommunity63%
MIT

Xihong Su, Jia Lin Hau, Gersi Doko, Kishan Panaganti, Marek Petrik. Risk-Averse Total-Reward Reinforcement Learning. The Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS 2025).

00JuliaOct 14, 20257 months agoMIT

Similar Papers

Risk-Averse Reinforcement Learning via Dynamic Time-Consistent Risk Measures

91% match2023

Risk-averse Total-reward MDPs with ERM and EVaR

90% match1 repo2024

Efficient Algorithms for Mitigating Uncertainty and Risk in Reinforcement Learning

89% match2025

Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs via Approximation by Discounted-Reward MDPs

86% match2024

A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents

86% match2024

Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processes

86% match1 repo2024

Cite this paper

@article{su2025riskaverse,
  title  = {Risk-Averse Total-Reward Reinforcement Learning},
  author = {Xihong Su and Jia Lin Hau and Gersi Doko and Kishan Panaganti and Marek Petrik},
  year   = {2025},
  eprint = {2506.21683},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.21683v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.21683.svg)](https://codeofpaper.com/paper/2506.21683)

Discussion