Harmonizing Stochasticity and Determinism: Scene-responsive Diverse Human Motion Prediction

Abstract

Diverse human motion prediction (HMP) is a fundamental application in computer vision that has recently attracted considerable interest. Prior methods primarily focus on the stochastic nature of human motion, while neglecting the specific impact of the external environment, leading to the pronounced artifacts in prediction when applied to real-world scenarios. To fill this gap, this work introduces a novel task: predicting diverse human motion within real-world 3D scenes. In contrast to prior works, it requires harmonizing the deterministic constraints imposed by the surrounding 3D scenes with the stochastic aspect of human motion. For this purpose, we propose DiMoP3D , a diverse motion prediction framework with 3D scene awareness, which leverages the 3D point cloud and observed sequence to generate diverse and high-fidelity predictions. DiMoP3D can comprehend the 3D scene and determine the probable target objects and their desired interactive pose based on the historical motion. Then, it plans the obstacle-free trajectories toward these interested objects and generates diverse and physically consistent future motions. On top of that, DiMoP3D identifies deterministic factors in the scene and integrates them into stochastic modeling, making the diverse HMP in realistic scenes become a controllable stochastic