Point Cloud Models Improve Visual Robustness in Robotic Learners

Skand Peri, Iain Lee, Chanho Kim, Li Fuxin, Tucker Hermans +1 more
4/29/2024
cs.ROcs.CVcs.LG

No official code found. Best implementation: MinchoU/pcwm-jax (0 stars, MIT, last commit 2025-05-26).

Abstract

Visual control policies can encounter significant performance degradation when visual conditions like lighting or camera position differ from those seen during training -- often exhibiting sharp declines in capability even for minor differences. In this work, we examine robustness to a suite of these types of visual changes for RGB-D and point cloud based visual control policies. To perform these experiments on both model-free and model-based reinforcement learners, we introduce a novel Point Cloud World Model (PCWM) and point cloud based control policies. Our experiments show that policies that explicitly encode point clouds are significantly more robust than their RGB-D counterparts. Further, we find our proposed PCWM significantly outperforms prior works in terms of sample efficiency during training. Taken together, these results suggest reasoning about the 3D scene through point clouds can improve performance, reduce learning time, and increase robustness for robotic learners. Project Webpage: https://pvskand.github.io/projects/PCWM

View on arXivView PDF

Code Implementations(1)

MinchoU/pcwm-jaxCommunity64%
MIT

Unofficial reproduction of PCWM (Point Cloud Models Improve Visual Robustness in Robotic Learners) based on original dreamer-v3 implementation.

00Shell, PythonMay 26, 20251 years agoMIT

Similar Papers

Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning

89% match2 repos912024

Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalization

89% match2025

On the Efficacy of 3D Point Cloud Reinforcement Learning

87% match1 repo402023

PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Clouds

85% match1 repo132024

GaussianPU: A Hybrid 2D-3D Upsampling Framework for Enhancing Color Point Clouds via 3D Gaussian Splatting

83% match2024

No Need for Real 3D: Fusing 2D Vision with Pseudo 3D Representations for Robotic Manipulation Learning

83% match2025

Cite this paper

@article{peri2024point,
  title  = {Point Cloud Models Improve Visual Robustness in Robotic Learners},
  author = {Skand Peri and Iain Lee and Chanho Kim and Li Fuxin and Tucker Hermans and Stefan Lee},
  year   = {2024},
  eprint = {2404.18926},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.18926v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.18926.svg)](https://codeofpaper.com/paper/2404.18926)

Discussion