A primal-dual perspective for distributed TD-learning

Han-Dong Lim, Donghwan Lee
10/1/2023
cs.LGmath.OC

Official code: limaries30/distributed-TD-learning (0 stars, last commit 2025-05-09).

Abstract

The goal of this paper is to investigate distributed temporal difference (TD) learning for a networked multi-agent Markov decision process. The proposed approach is based on distributed optimization algorithms, which can be interpreted as primal-dual Ordinary differential equation (ODE) dynamics subject to null-space constraints. Based on the exponential convergence behavior of the primal-dual ODE dynamics subject to null-space constraints, we examine the behavior of the final iterate in various distributed TD-learning scenarios, considering both constant and diminishing step-sizes and incorporating both i.i.d. and Markovian observation models. Unlike existing methods, the proposed algorithm does not require the assumption that the underlying communication network structure is characterized by a doubly stochastic matrix.

View on arXivView PDF

Code Implementations(1)

limaries30/distributed-TD-learningOfficial
00Python, Jupyter NotebookApr 29, 20251 years ago

Similar Papers

Distributed TD(0) with Almost No Communication

84% match2023

Personalized Multi-Agent Average Reward TD-Learning via Joint Linear Approximation

81% match2026

Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneity

81% match2023

Discretized Distributed Optimization over Dynamic Digraphs

81% match2023

Accelerating Distributed Optimization: A Primal-Dual Perspective on Local Steps

81% match2024

Asynchronous Message-Passing and Zeroth-Order Optimization Based Distributed Learning with a Use-Case in Resource Allocation in Communication Networks

80% match2023

Cite this paper

@article{lim2023primaldual,
  title  = {A primal-dual perspective for distributed TD-learning},
  author = {Han-Dong Lim and Donghwan Lee},
  year   = {2023},
  eprint = {2310.00638},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.00638v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.00638.svg)](https://codeofpaper.com/paper/2310.00638)

Discussion