HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting

Abstract

We have recently seen tremendous progress in photo-real human modeling and rendering. Yet, efficiently ren-dering realistic human performance and integrating it into the rasterization pipeline remains challenging. In this pa-per, we present HiFi4G, an explicit and compact Gaussian-based approach for high-fidelity human performance ren-dering from dense footage. Our core intuition is to marry the 3D Gaussian representation with non-rigid tracking, achieving a compact and compression-friendly representation. We first propose a dual-graph mechanism to obtain motion priors, with a coarse deformation graph for effective initialization and a fine-grained Gaussian graph to en-force subsequent constraints. Then, we utilize a 4D Gaus-sian optimization scheme with adaptive spatial-temporal regularizers to effectively balance the non-rigid prior and Gaussian updating. We also present a companion compression scheme with residual compensation for immersive experiences on various platforms. It achieves a substantial compression rate of approximately 25 times, with less than 2MB of storage per frame. Extensive experiments demon-strate the effectiveness of our approach, which significantly outperforms existing approaches in terms of optimization speed, rendering quality, and storage overhead. Project page: https://nowheretrix.github.io/HiFi4G/.