Diversity Is Not All You Need: Training A Robust Cooperative Agent Needs Specialist Partners

Abstract

Partner diversity is known to be crucial for training a robust generalist cooperative agent. In this paper, we show that partner specialization, in addition to diversity, is crucial for the robustness of a downstream generalist agent. We propose a principled method for quantifying both the diversity and specialization of a partner population based on the concept of mutual information. Then, we observe that the recently proposed cross-play minimization ( XP-min ) technique produces diverse and specialized partners. However, the generated partners are overfit , reducing their usefulness as training partners. To address this, we propose simple methods, based on reinforcement learning and supervised learning, for extracting the diverse and specialized behaviors of XP-min generated partners but not their overfitness. We demonstrate empirically that the proposed method effectively removes overfitness, and extracted populations produce more robust generalist agents compared to the source XP-min populations. This result highlights the importance of considering both the diversity and specialization of training partners while carefully managing their overfitness for training robust cooperative generalists.