Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Planner

Kenneth Li, Yiming Wang, Fernanda Viégas, Martin Wattenberg
6/17/2024
cs.CLcs.AIcs.LG

No official code found. Best implementation: likenneth/dialogue_action_token (31 stars, MIT, last commit 2024-06-27).

Abstract

We present an approach called Dialogue Action Tokens (DAT) that adapts language model agents to plan goal-directed dialogues. The core idea is to treat each utterance as an action, thereby converting dialogues into games where existing approaches such as reinforcement learning can be applied. Specifically, we freeze a pretrained language model and train a small planner model that predicts a continuous action vector, used for controlled generation in each round. This design avoids the problem of language degradation under reward optimization. When evaluated on the Sotopia platform for social simulations, the DAT-steered LLaMA model surpasses GPT-4's performance. We also apply DAT to steer an attacker language model in a novel multi-turn red-teaming setting, revealing a potential new attack surface.

View on arXivView PDF

Code Implementations(1)

likenneth/dialogue_action_tokenCommunity100%
MIT

Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Planner

311PythonJun 6, 20242 years agoMIT

Similar Papers

LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models

87% match1 repo1172026

GameTalk: Training LLMs for Strategic Conversation

87% match2026

Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents

85% match1 repo332026

Playpen: An Environment for Exploring Learning Through Conversational Interaction

85% match1 repo142025

Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game

84% match2023

Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game

84% match2026

Cite this paper

@article{li2024dialogue,
  title  = {Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Planner},
  author = {Kenneth Li and Yiming Wang and Fernanda Viégas and Martin Wattenberg},
  year   = {2024},
  eprint = {2406.11978},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.11978v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.11978.svg)](https://codeofpaper.com/paper/2406.11978)

Discussion