Colour versus Shape Goal Misgeneralization in Reinforcement Learning: A Case Study

Karolis Ramanauskas, Özgür Şimşek
12/5/2023
cs.LGcs.AI

No official code found. Best implementation: KarolisRam/colour-shape-goal-misgeneralization (1 star, MIT, last commit 2025-03-28).

Abstract

We explore colour versus shape goal misgeneralization originally demonstrated by Di Langosco et al. (2022) in the Procgen Maze environment, where, given an ambiguous choice, the agents seem to prefer generalization based on colour rather than shape. After training over 1,000 agents in a simplified version of the environment and evaluating them on over 10 million episodes, we conclude that the behaviour can be attributed to the agents learning to detect the goal object through a specific colour channel. This choice is arbitrary. Additionally, we show how, due to underspecification, the preferences can change when retraining the agents using exactly the same procedure except for using a different random seed for the training run. Finally, we demonstrate the existence of outliers in out-of-distribution behaviour based on training random seed alone.

View on arXivView PDF

Code Implementations(1)

KarolisRam/colour-shape-goal-misgeneralizationCommunity100%
MIT

Code for paper "Colour versus Shape Goal Misgeneralization in Reinforcement Learning: A Case Study"

10Shell, PythonNov 16, 20231 years agoMIT

Similar Papers

Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learning

82% match1 repo132022

Mitigating Goal Misgeneralization via Minimax Regret

81% match2025

Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning

80% match2026

Bootstrap State Representation using Style Transfer for Better Generalization in Deep Reinforcement Learning

79% match2022

Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learning

79% match2025

Disentangling Recognition and Decision Regrets in Image-Based Reinforcement Learning

79% match1 repo12024

Cite this paper

@article{ramanauskas2023colour,
  title  = {Colour versus Shape Goal Misgeneralization in Reinforcement Learning: A Case Study},
  author = {Karolis Ramanauskas and Özgür Şimşek},
  year   = {2023},
  eprint = {2312.03762},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.03762v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.03762.svg)](https://codeofpaper.com/paper/2312.03762)

Discussion