Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

Michelle A. Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese +3 more
10/24/2018
cs.ROcs.AIcs.LG

Official code: stanford-iprl-lab/multimodal_representation (103 stars, MIT, last commit 2023-01-01).

Abstract

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. We evaluate our method on a peg insertion task, generalizing over different geometry, configurations, and clearances, while being robust to external perturbations. Results for simulated and real robot experiments are presented.

View on arXivView PDF

Code Implementations(1)

stanford-iprl-lab/multimodal_representationOfficial100%
MIT
10321Shell, PythonMay 12, 20203 years agoMIT

Similar Papers

Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data

87% match1 repo222023

Bi-Manual Block Assembly via Sim-to-Real Reinforcement Learning

87% match2023

Learning to Design and Use Tools for Robotic Manipulation

86% match2023

Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning

86% match2023

Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning

85% match1 repo1.1k2018

Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration

85% match2025

Cite this paper

@article{lee2018making,
  title  = {Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks},
  author = {Michelle A. Lee and Yuke Zhu and Krishnan Srinivasan and Parth Shah and Silvio Savarese and Li Fei-Fei and Animesh Garg and Jeannette Bohg},
  year   = {2018},
  eprint = {1810.10191},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1810.10191}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1810.10191.svg)](https://codeofpaper.com/paper/1810.10191)

Discussion