Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions

Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia +20 more
9/18/2023
cs.ROcs.AIcs.LG

No official code found. Best implementation: lucidrains/q-transformer (407 stars, MIT, last commit 2025-06-20).

Abstract

In this work, we present a scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both human demonstrations and autonomously collected data. Our method uses a Transformer to provide a scalable representation for Q-functions trained via offline temporal difference backups. We therefore refer to the method as Q-Transformer. By discretizing each action dimension and representing the Q-value of each action dimension as separate tokens, we can apply effective high-capacity sequence modeling techniques for Q-learning. We present several design decisions that enable good performance with offline RL training, and show that Q-Transformer outperforms prior offline RL algorithms and imitation learning techniques on a large diverse real-world robotic manipulation task suite. The project's website and videos can be found at https://qtransformer.github.io

View on arXivView PDF

Code Implementations(1)

lucidrains/q-transformerCommunity74%
MIT

Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind

40722PythonSep 20, 20231 years agoMIT
artificial-intelligenceattention-mechanismsdeep-learningoffline-learningq-learning+2 more

Similar Papers

Q-value Regularized Decision ConvFormer for Offline Reinforcement Learning

86% match2024

QT-TDM: Planning With Transformer Dynamics Model and Autoregressive Q-Learning

86% match2 repos62024

When should we prefer Decision Transformers for Offline Reinforcement Learning?

85% match1 repo172023

Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning

85% match2024

TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse

84% match2026

In-Context Compositional Q-Learning for Offline Reinforcement Learning

84% match2025

Cite this paper

@article{chebotar2023qtransformer,
  title  = {Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions},
  author = {Yevgen Chebotar and Quan Vuong and Alex Irpan and Karol Hausman and Fei Xia and Yao Lu and Aviral Kumar and Tianhe Yu and Alexander Herzog and Karl Pertsch and Keerthana Gopalakrishnan and Julian Ibarz and Ofir Nachum and Sumedh Sontakke and Grecia Salazar and Huong T Tran and Jodilyn Peralta and Clayton Tan and Deeksha Manjunath and Jaspiar Singht and Brianna Zitkovich and Tomas Jackson and Kanishka Rao and Chelsea Finn and Sergey Levine},
  year   = {2023},
  eprint = {2309.10150},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.10150v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.10150.svg)](https://codeofpaper.com/paper/2309.10150)

Discussion