Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs

Haonan Chang, Kowndinya Boyalakuntla, Shiyang Lu, Siwei Cai, Eric Jing +6 more
9/27/2023
cs.ROcs.CV

No official code found. Best implementation: changhaonan/OVSG (73 stars, last commit 2023-12-30).

Abstract

We present an Open-Vocabulary 3D Scene Graph (OVSG), a formal framework for grounding a variety of entities, such as object instances, agents, and regions, with free-form text-based queries. Unlike conventional semantic-based object localization approaches, our system facilitates context-aware entity localization, allowing for queries such as ``pick up a cup on a kitchen table" or ``navigate to a sofa on which someone is sitting". In contrast to existing research on 3D scene graphs, OVSG supports free-form text input and open-vocabulary querying. Through a series of comparative experiments using the ScanNet dataset and a self-collected dataset, we demonstrate that our proposed approach significantly surpasses the performance of previous semantic-based localization techniques. Moreover, we highlight the practical application of OVSG in real-world robot navigation and manipulation experiments.

View on arXivView PDF

Code Implementations(1)

changhaonan/OVSGCommunity100%

[CoRL2023] Open-Vocabulary Scene-Graph

734PythonSep 24, 20232 years ago

Similar Papers

Instance-Enriched Semantic Maps for Visual Language Navigation

83% match2 repos2026

Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models

83% match1 repo382024

Semantic Environment Atlas for Object-Goal Navigation

83% match2024

Agentic Scene Policies: Unifying Space, Semantics, and Affordances for Robot Action

83% match2025

"Where am I?" Scene Retrieval with Language

83% match2026

Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph

83% match2024

Cite this paper

@article{chang2023contextaware,
  title  = {Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs},
  author = {Haonan Chang and Kowndinya Boyalakuntla and Shiyang Lu and Siwei Cai and Eric Jing and Shreesh Keskar and Shijie Geng and Adeeb Abbas and Lifeng Zhou and Kostas Bekris and Abdeslam Boularias},
  year   = {2023},
  eprint = {2309.15940},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.15940v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.15940.svg)](https://codeofpaper.com/paper/2309.15940)

Discussion