A Prospect-Theoretic Policy Gradient Framework for Behaviorally Nuanced Reinforcement Learning

Abstract

Classical reinforcement learning (RL) typically assumes rational decision-making based on expected utility theory. However, this model has been shown to be empirically inconsistent with actual human preferences, as evidenced in psychology and behavioral economics. Cumulative Prospect Theory (CPT) provides a more nuanced model for human-based decision-making, capturing diverse attitudes and perceptions toward risk, gains, and losses. While prior work has integrated CPT with RL to solve CPT policy optimization problems, the understanding and impact of this formulation remain limited. Our contributions are as follows: (a) we derive a novel policy gradient theorem for CPT objectives, generalizing the foundational result in standard RL, (b) we design a model-free policy gradient algorithm for solving the CPT-RL problem, (c) we analyze our policy gradient estimator and prove asymptotic convergence of the algorithm to first-order stationary points, and (d) test its performance through simulations. Notably, our first-order policy gradient algorithm scales better than existing zeroth-order methods to larger state spaces. Our theoretical framework offers more flexibility to advance the integration of behavioral decision-making into RL.