CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics

David Valencia, Henry Williams, Yuning Xing, Trevor Gee, Bruce A MacDonald +1 more
5/4/2024
cs.LGcs.AI

No official code found. Best implementation: UoA-CARES/cares_reinforcement_learning (41 stars, MIT, last commit 2026-05-01).

Abstract

Categorical Distributional Reinforcement Learning (CDRL) has demonstrated superior sample efficiency in learning complex tasks compared to conventional Reinforcement Learning (RL) approaches. However, the practical application of CDRL is encumbered by challenging projection steps, detailed parameter tuning, and domain knowledge. This paper addresses these challenges by introducing a pioneering Continuous Distributional Model-Free RL algorithm tailored for continuous action spaces. The proposed algorithm simplifies the implementation of distributional RL, adopting an actor-critic architecture wherein the critic outputs a continuous probability distribution. Additionally, we propose an ensemble of multiple critics fused through a Kalman fusion mechanism to mitigate overestimation bias. Through a series of experiments, we validate that our proposed method provides a sample-efficient solution for executing complex continuous-control tasks.

View on arXivView PDF

Code Implementations(1)

UoA-CARES/cares_reinforcement_learningCommunity100%
MIT

CARES Reinforcement Learning Package

4113PythonNov 17, 20225 months agoMIT
deep-learningmachine-learningreinforcement-learningrobotics

Similar Papers

Distribution Parameter Actor-Critic: Shifting the Agent-Environment Boundary for Diverse Action Spaces

87% match2025

Continuous Control Reinforcement Learning: Distributed Distributional DrQ Algorithms

85% match2024

Implicit Distributional Reinforcement Learning

85% match1 repo122020

D2 Actor Critic: Diffusion Actor Meets Distributional Critic

84% match2025

Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning

84% match1 repo222025

Distributional Soft Actor-Critic with Three Refinements

83% match1 repo4432023

Cite this paper

@article{valencia2024ctd,
  title  = {CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics},
  author = {David Valencia and Henry Williams and Yuning Xing and Trevor Gee and Bruce A MacDonald and Minas Liarokapis},
  year   = {2024},
  eprint = {2405.02576},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.02576v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.02576.svg)](https://codeofpaper.com/paper/2405.02576)

Discussion