UniPAD: A Universal Pre-Training Paradigm for Autonomous Driving

Abstract

In the context of autonomous driving, the significance of effective feature learning is widely acknowledged. While conventional 3D self-supervised pretraining methods have shown widespread success, most methods follow the ideas originally designed for 2D images. In this paper, we present UniPAD, a novel self-supervised learning paradigm applying 3D volumetric differentiable rendering. UniPAD implicitly encodes 3D space, facilitating the reconstruction of continuous 3D shape structures and the intricate appear-ance characteristics of their 2D projections. The flexibil-ity of our method enables seamless integration into both 2D and 3D frameworks, enabling a more holistic compre-hension of the scenes. We manifest the feasibility and effectiveness of UniPAD by conducting extensive experiments on various 3D perception tasks. Our method significantly improves lidar-, camera-, and lidar-camera-based baseline by 9.1, 7.7, and 6.9 NDS, respectively. Notably, our pretraining pipeline achieves 73.2 NDS for 3D object detection and 79.4 mIoU for 3D semantic segmentation on the nuScenes validation set, achieving state-of-the-art results in comparison with previous methods.