For SALE: State-Action Representation Learning for Deep Reinforcement Learning

Scott Fujimoto, Wei-Di Chang, Edward J. Smith, Shixiang Shane Gu, Doina Precup +1 more
6/4/2023
cs.LGcs.AIstat.ML

No official code found. Best implementation: sfujim/TD7 (170 stars, MIT, last commit 2023-09-12).

Abstract

In the field of reinforcement learning (RL), representation learning is a proven tool for complex image-based tasks, but is often overlooked for environments with low-level states, such as physical control problems. This paper introduces SALE, a novel approach for learning embeddings that model the nuanced interaction between state and action, enabling effective representation learning from low-level states. We extensively study the design space of these embeddings and highlight important design considerations. We integrate SALE and an adaptation of checkpoints for RL into TD3 to form the TD7 algorithm, which significantly outperforms existing continuous control algorithms. On OpenAI gym benchmark tasks, TD7 has an average performance gain of 276.7% and 50.7% over TD3 at 300k and 5M time steps, respectively, and works in both the online and offline settings.

View on arXivView PDF

Code Implementations(1)

sfujim/TD7Community100%
MIT

Author's PyTorch implementation of TD7 for online and offline RL

17013PythonJun 4, 20233 years agoMIT

Similar Papers

A Survey of State Representation Learning for Deep Reinforcement Learning

88% match2025

LLM-Empowered State Representation for Reinforcement Learning

85% match1 repo422026

Representations and Exploration for Deep Reinforcement Learning using Singular Value Decomposition

85% match2023

Unsupervised Representation Learning in Partially Observable Atari Games

85% match2023

Bootstrapped Representations in Reinforcement Learning

83% match2023

Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions

83% match2025

Cite this paper

@article{fujimoto2023sale,
  title  = {For SALE: State-Action Representation Learning for Deep Reinforcement Learning},
  author = {Scott Fujimoto and Wei-Di Chang and Edward J. Smith and Shixiang Shane Gu and Doina Precup and David Meger},
  year   = {2023},
  eprint = {2306.02451},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.02451v2},
  journal = {NEURIPS 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.02451.svg)](https://codeofpaper.com/paper/2306.02451)

Discussion