Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments

Abstract

Research into dynamic 3D scene understanding has pri-marily focused on short-term change tracking from dense observations, while little attention has been paid to long-term changes with sparse observations. We address this gap with More2,a novel approach for multi-object relo-calization and reconstruction in evolving environments. We view these environments as “living scenes” and consider the problem of transforming scans taken at different points in time into a 3D reconstruction of the object instances, whose accuracy and completeness increase over time. At the core of our method lies an SE (3)-equivariant represen-tation in a single encoder-decoder network, trained on syn-thetic data. This representation enables us to seamlessly tackle instance matching, registration, and reconstruction. We also introduce a joint optimization algorithm that facil-itates the accumulation of point clouds originating from the same instance across multiple scans taken at different points in time. We validate our method on synthetic and real-world data and demonstrate state-of-the-art performance in both end-to-end performance and individual subtasks. [project]