Risk-averse Total-reward MDPs with ERM and EVaR

Xihong Su, Julien Grand-Clément, Marek Petrik
8/30/2024
cs.LGcs.AI

No official code found. Best implementation: suxh2019/ERMLP (0 stars, MIT, last commit 2026-03-23).

Abstract

Optimizing risk-averse objectives in discounted MDPs is challenging because most models do not admit direct dynamic programming equations and require complex history-dependent policies. In this paper, we show that the risk-averse {\em total reward criterion}, under the Entropic Risk Measure (ERM) and Entropic Value at Risk (EVaR) risk measures, can be optimized by a stationary policy, making it simple to analyze, interpret, and deploy. We propose exponential value iteration, policy iteration, and linear programming to compute optimal policies. Compared with prior work, our results only require the relatively mild condition of transient MDPs and allow for {\em both} positive and negative rewards. Our results indicate that the total reward criterion may be preferable to the discounted criterion in a broad range of risk-averse reinforcement learning domains.

View on arXivView PDF

Code Implementations(1)

suxh2019/ERMLPCommunity100%
MIT

Risk-averse Total-reward MDPs with ERM and EVaR. The 39th Annual AAAI Conference on Artificial Intelligence(AAAI2025))

00JuliaDec 12, 20247 months agoMIT

Similar Papers

Risk-Averse Total-Reward Reinforcement Learning

90% match1 repo2025

Efficient Algorithms for Mitigating Uncertainty and Risk in Reinforcement Learning

87% match2025

Risk-Averse Reinforcement Learning via Dynamic Time-Consistent Risk Measures

86% match2023

A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents

85% match2024

Risk-sensitive Markov Decision Process and Learning under General Utility Functions

85% match2023

Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs via Approximation by Discounted-Reward MDPs

84% match2024

Cite this paper

@article{su2024riskaverse,
  title  = {Risk-averse Total-reward MDPs with ERM and EVaR},
  author = {Xihong Su and Julien Grand-Clément and Marek Petrik},
  year   = {2024},
  eprint = {2408.17286},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.17286v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.17286.svg)](https://codeofpaper.com/paper/2408.17286)

Discussion