HuBo-VLM: Unified Vision-Language Model designed for HUman roBOt interaction tasks

Zichao Dong, Weikun Zhang, Xufeng Huang, Hang Ji, Xin Zhan +1 more
8/24/2023
cs.ROcs.CV

Official code: dzcgaara/HuBo-VLM (7 stars, last commit 2023-08-24).

Abstract

Human robot interaction is an exciting task, which aimed to guide robots following instructions from human. Since huge gap lies between human natural language and machine codes, end to end human robot interaction models is fair challenging. Further, visual information receiving from sensors of robot is also a hard language for robot to perceive. In this work, HuBo-VLM is proposed to tackle perception tasks associated with human robot interaction including object detection and visual grounding by a unified transformer based vision language model. Extensive experiments on the Talk2Car benchmark demonstrate the effectiveness of our approach. Code would be publicly available in https://github.com/dzcgaara/HuBo-VLM.

View on arXivView PDF

Code Implementations(1)

dzcgaara/HuBo-VLMOfficial100%

Official implementation of HuBo-VLM.

70Aug 24, 20233 years ago

Similar Papers

WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model

88% match2023

GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication

85% match2025

Evo-1: Lightweight Vision-Language-Action Model with Preserved Semantic Alignment

85% match1 repo3382025

ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understanding

84% match2023

RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics

84% match1 repo2302024

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

84% match2024

Cite this paper

@article{dong2023hubovlm,
  title  = {HuBo-VLM: Unified Vision-Language Model designed for HUman roBOt interaction tasks},
  author = {Zichao Dong and Weikun Zhang and Xufeng Huang and Hang Ji and Xin Zhan and Junbo Chen},
  year   = {2023},
  eprint = {2308.12537},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.12537v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.12537.svg)](https://codeofpaper.com/paper/2308.12537)

Discussion