Decision Transformer: Reinforcement Learning via Sequence Modeling

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover +4 more
6/2/2021
cs.LGcs.AI

Official code: kzl/decision-transformer (2,844 stars, MIT, last commit 2024-04-29).

Abstract

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

View on arXivView PDF

Code Implementations(1)

kzl/decision-transformerOfficial100%
MIT

Official codebase for Decision Transformer: Reinforcement Learning via Sequence Modeling.

2,844517Shell, PythonJun 2, 20212 years agoMIT

Similar Papers

Efficiently Modeling Long Sequences with Structured State Spaces

83% match1 repo2.9k2021

MambaLRP: Explaining Selective State Space Sequence Models

82% match1 repo472024

Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks

82% match1 repo1252017

On Subquadratic Architectures: From Applications to Principles

81% match2026

Discrete Autoencoders for Sequence Models

81% match1 repo17.5k2018

Rethinking Decision Transformer via Hierarchical Reinforcement Learning

80% match2023

Cite this paper

@article{chen2021decision,
  title  = {Decision Transformer: Reinforcement Learning via Sequence Modeling},
  author = {Lili Chen and Kevin Lu and Aravind Rajeswaran and Kimin Lee and Aditya Grover and Michael Laskin and Pieter Abbeel and Aravind Srinivas and Igor Mordatch},
  year   = {2021},
  eprint = {2106.01345},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2106.01345}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2106.01345.svg)](https://codeofpaper.com/paper/2106.01345)

Discussion