Policy Learning from Tutorial Books via Understanding, Rehearsing and Introspecting

Abstract

When humans need to learn a new skill, we can acquire knowledge through written books, including textbooks, tutorials, etc. However, current research for decision-making, like reinforcement learning (RL), has primarily required numerous real interactions with the target environment to learn a skill, while failing to utilize the existing knowledge already summarized in the text. The success of Large Language Models (LLMs) sheds light on utilizing such knowledge behind the books. In this paper, we discuss a new policy learning problem called P olicy L earning from tutorial B ooks (PLfB) upon the shoulders of LLMs’ systems, which aims to leverage rich resources such as tutorial books to derive a policy network. Inspired by how humans learn from books, we solve the problem via a three-stage framework: U nderstanding, R ehearsing, and I ntrospecting (URI). In particular, it first rehearses decision-making trajectories based on the derived knowledge after understanding the books, then introspects about the imaginary dataset to distill a policy network. We build two benchmarks for PLfB based on Tic-Tac-Toe and Football games. In the experiment, URI’s policy achieves a minimum of 44% net winning rate against GPT-based agents without any real data. In the much more complex football game, URI’s policy beat the built-in AIs with a 37% winning rate while GPT-based agents can only achieve a 6% winning rate. The project page