Central Limit Theorems for Asynchronous Averaged Q-Learning

Xingtu Liu
9/23/2025
cs.LGmath.OCstat.ML

No confident code implementation found for this paper.

Abstract

This paper establishes central limit theorems for Polyak-Ruppert averaged Q-learning under asynchronous updates. We prove a non-asymptotic central limit theorem, where the convergence rate in Wasserstein distance explicitly reflects the dependence on the number of iterations, state-action space size, the discount factor, and the quality of exploration. In addition, we derive a functional central limit theorem, showing that the partial-sum process converges weakly to a Brownian motion.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Non-Asymptotic Guarantees for Average-Reward Q-Learning with Adaptive Stepsizes

84% match2025

Rates of Convergence in the Central Limit Theorem for Markov Chains, with an Application to TD Learning

81% match2024

Rates of Convergence in the Central Limit Theorem for Markov Chains, with an Application to TD Learning

81% match2024

Average-reward reinforcement learning in semi-Markov decision processes via relative value iteration

81% match2025

Asynchronous Stochastic Approximation with Applications to Average-Reward Reinforcement Learning

80% match2024

Prelimit Coupling and Steady-State Convergence of Constant-stepsize Nonsmooth Contractive SA

79% match2024

Cite this paper

@article{liu2025central,
  title  = {Central Limit Theorems for Asynchronous Averaged Q-Learning},
  author = {Xingtu Liu},
  year   = {2025},
  eprint = {2509.18964},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2509.18964v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2509.18964.svg)](https://codeofpaper.com/paper/2509.18964)

Discussion