Distributed TD(0) with Almost No Communication

Rui Liu, Alex Olshevsky
5/25/2023
cs.LGeess.SYmath.OC

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

We provide a new non-asymptotic analysis of distributed temporal difference learning with linear function approximation. Our approach relies on ``one-shot averaging,'' where $N$ agents run identical local copies of the TD(0) method and average the outcomes only once at the very end. We demonstrate a version of the linear time speedup phenomenon, where the convergence time of the distributed process is a factor of $N$ faster than the convergence time of TD(0). This is the first result proving benefits from parallelism for temporal difference methods.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

A primal-dual perspective for distributed TD-learning

84% match1 repo2023

One-Shot Averaging for Distributed TD(λ) Under Markov Sampling

84% match2024

DASA: Delay-Adaptive Multi-Agent Stochastic Approximation

83% match2024

Accelerated Distributional Temporal Difference Learning with Linear Function Approximation

82% match2025

A Finite-Time Analysis of TD Learning with Linear Function Approximation without Projections or Strong Convexity

82% match2025

Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes

81% match2026

Cite this paper

@article{liu2023distributed,
  title  = {Distributed TD(0) with Almost No Communication},
  author = {Rui Liu and Alex Olshevsky},
  year   = {2023},
  eprint = {2305.16246},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.16246v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.16246.svg)](https://codeofpaper.com/paper/2305.16246)

Discussion