SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion

Abstract

A long-standing goal of 3D human reconstruction is to cre-ate lifelike and fully detailed 3D humans from single-view images. The main challenge lies in inferring unknown body shapes, appearances, and clothing details in areas not visi-ble in the images. To address this, we propose SiTH, a novel pipeline that uniquely integrates an image-conditioned dif-fusion model into a 3D mesh reconstruction workflow. At the core of our method lies the decomposition of the chal-lenging single-view reconstruction problem into generative hallucination and reconstruction subproblems. For the for-mer, we employ a powerful generative diffusion model to hallucinate unseen back-view appearance based on the in-put images. For the latter, we leverage skinned body meshes as guidance to recover full-body texture meshes from the in-put and back-view images. SiTH requires as few as 500 3D human scans for training while maintaining its generality and robustness to diverse images. Extensive evaluations on two 3D human benchmarks, including our newly created one, highlighted our method's superior accuracy and per-ceptual quality in 3D textured human reconstruction.