Meta-Q-Learning

Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, Alexander J. Smola
9/30/2019
cs.LGstat.ML

Official code: amazon-science/meta-q-learning (108 stars, last commit 2022-06-18).

Abstract

This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-Reinforcement Learning (meta-RL). MQL builds upon three simple ideas. First, we show that Q-learning is competitive with state-of-the-art meta-RL algorithms if given access to a context variable that is a representation of the past trajectory. Second, a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies. Third, past data from the meta-training replay buffer can be recycled to adapt the policy on a new task using off-policy updates. MQL draws upon ideas in propensity estimation to do so and thereby amplifies the amount of available data for adaptation. Experiments on standard continuous-control benchmarks suggest that MQL compares favorably with the state of the art in meta-RL.

View on arXivView PDF

Code Implementations(1)

amazon-science/meta-q-learningOfficial100%

Code for the paper "Meta-Q-Learning"( ICLR 2020)

10818Ruby, ShellSep 2, 20204 years agoNOASSERTION
deep-learningmeta-learningmulti-task-learningreinforcement-learning

Similar Papers

Learning Action Translator for Meta Reinforcement Learning on Sparse-Reward Tasks

88% match2022

Meta Reinforcement Learning with Successor Feature Based Context

86% match2022

Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables

86% match1 repo5152019

Improving Generalization of Meta-Learning with Inverted Regularization at Inner-Level

84% match2023

Meta-Learning with Versatile Loss Geometries for Fast Adaptation Using Mirror Descent

84% match2023

PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees

84% match1 repo262020

Cite this paper

@article{fakoor2019metaqlearning,
  title  = {Meta-Q-Learning},
  author = {Rasool Fakoor and Pratik Chaudhari and Stefano Soatto and Alexander J. Smola},
  year   = {2019},
  eprint = {1910.00125},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1910.00125}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1910.00125.svg)](https://codeofpaper.com/paper/1910.00125)

Discussion