Twinner: Shining Light on Digital Twins in a Few Snaps

Abstract

We present Twinner, the first large reconstruction model capable of recovering a scene’s illumination as well as an object’s geometry and material properties from only a few posed images. Twinner is based on the Large Reconstruction Model and innovates in three key ways: 1)We introduce a memory-efficient voxel-grid transformer whose memory scales only quadratically with the size of the voxel grid. 2) To address the scarcity of high-quality ground-truth PBR- shaded models, we introduce a large, fully synthetic dataset of procedurally generated PBR-textured objects lit with varied illumination. 3) To bridge the synthetic-to-real gap, we finetune the model on real-world datasets using a differentiable physically based shading model, eliminating the need for ground-truth illumination or material properties, which are challenging to obtain in real-world scenarios. We demonstrate the efficacy of our model on the real-world StanfordORB benchmark, where, given a few input views, we achieve reconstruction quality significantly superior to existing feed-forward reconstruction networks and comparable to slower per-scene optimization methods.