Relightable Gaussian Codec Avatars

Abstract

The fidelity of relighting is bounded by both geometry and appearance representations. For geometry, both mesh and volumetric approaches have difficulty modeling intri-cate structures like 3D hair geometry. For appearance, existing relighting models are limited in fidelity and often too slow to render in real-time with high-resolution contin-uous environments. In this work, we present Relightable Gaussian Codec Avatars, a method to build high-fidelity relightable head avatars that can be animated to generate novel expressions. Our geometry model based on 3D Gaus-sians can capture 3D-consistent sub-millimeter details such as hair strands and pores on dynamic face sequences. To support diverse materials of human heads such as the eyes, skin, and hair in a unified manner, we present a novel re-lightable appearance model based on learnable radiance transfer. Together with global illumination-aware spheri-cal harmonics for the diffuse components, we achieve real-time relighting with all-frequency reflections using spheri-cal Gaussians. This appearance model can be efficiently relit under both point light and continuous illumination. We further improve the fidelity of eye reflections and enable ex-plicit gaze control by introducing relightable explicit eye models. Our method outperforms existing approaches with-out compromising real-time performance. We also demon-strate real-time relighting of avatars on a tethered con-sumer VR headset, showcasing the efficiency and fidelity of our avatars.