Visual-Tactile Sensing for In-Hand Object Reconstruction

Abstract

Tactile sensing is one of the modalities humans rely on heavily to perceive the world. Working with vision, this modality refines local geometry structure, measures defor-mation at the contact area, and indicates the hand-object contact state. With the availability of open-source tactile sensors such as DIGIT, research on visual-tactile learning is becoming more accessible and reproducible. Leveraging this tactile sensor, we propose a novel visual-tactile in-hand object reconstruction framework VTacO, and ex-tend it to VTacOH for hand-object reconstruction. Since our method can support both rigid and deformable ob-ject reconstruction, no existing benchmarks are proper for the goal. We propose a simulation environment, VT-Sim, which supports generating hand-object interaction for both rigid and deformable objects. With VT-Sim, we gener-ate a large-scale training dataset and evaluate our method on it. Extensive experiments demonstrate that our pro-posed method can outperform the previous baseline meth-ods qualitatively and quantitatively. Finally, we directly ap-ply our model trained in simulation to various real-world test cases, which display qualitative results. Codes, mod-els, simulation environment, and datasets are available at https://sites.google.com/view/vtaco/.