VHAKG: A Multi-modal Knowledge Graph Based on Synchronized Multi-view Videos of Daily Activities

Shusaku Egami, Takahiro Ugai, Swe Nwe Nwe Htun, Ken Fukuda
8/27/2024
cs.AIcs.CLcs.CV

No official code found. Best implementation: aistairc/virtualhome_aist (4 stars, last commit 2025-01-08). 2 more below.

Abstract

Multi-modal knowledge graphs (MMKGs), which ground various non-symbolic data (e.g., images and videos) into symbols, have attracted attention as resources enabling knowledge processing and machine learning across modalities. However, the construction of MMKGs for videos consisting of multiple events, such as daily activities, is still in the early stages. In this paper, we construct an MMKG based on synchronized multi-view simulated videos of daily activities. Besides representing the content of daily life videos as event-centric knowledge, our MMKG also includes frame-by-frame fine-grained changes, such as bounding boxes within video frames. In addition, we provide support tools for querying our MMKG. As an application example, we demonstrate that our MMKG facilitates benchmarking vision-language models by providing the necessary vision-language datasets for a tailored task.

View on arXivView PDF

Code Implementations(3)

aistairc/virtualhome_aistCommunity100%
41Shell, PythonNov 5, 20211 years agoNOASSERTION
aistairc/vhakg-toolsCommunity100%
MIT

Support tools for VHAKG (VirtualHome-AIST-KG)

11CSS, HTMLMay 30, 20241 years agoMIT
aistairc/vhakgCommunity72%

Multi-modal Knowledge Graphs with Multi-view Videos of Daily Activities

10PythonJun 18, 20241 years ago

Similar Papers

From Videos to Indexed Knowledge Graphs -- Framework to Marry Methods for Multimodal Content Analysis and Understanding

86% match2025

VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings

83% match2026

VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation

82% match2025

Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videos

82% match2026

Asynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations

81% match2024

NativE: Multi-modal Knowledge Graph Completion in the Wild

81% match1 repo552024

Cite this paper

@article{egami2024vhakg,
  title  = {VHAKG: A Multi-modal Knowledge Graph Based on Synchronized Multi-view Videos of Daily Activities},
  author = {Shusaku Egami and Takahiro Ugai and Swe Nwe Nwe Htun and Ken Fukuda},
  year   = {2024},
  eprint = {2408.14895},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.14895v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.14895.svg)](https://codeofpaper.com/paper/2408.14895)

Discussion