ViLP: Knowledge Exploration using Vision, Language, and Pose Embeddings for Video Action Recognition

Soumyabrata Chaudhuri, Saumik Bhattacharya
8/7/2023
cs.CVcs.AIcs.LG

Official code: Soumyabrata2003/ViLP (5 stars, MIT, last commit 2024-10-08).

Abstract

Video Action Recognition (VAR) is a challenging task due to its inherent complexities. Though different approaches have been explored in the literature, designing a unified framework to recognize a large number of human actions is still a challenging problem. Recently, Multi-Modal Learning (MML) has demonstrated promising results in this domain. In literature, 2D skeleton or pose modality has often been used for this task, either independently or in conjunction with the visual information (RGB modality) present in videos. However, the combination of pose, visual information, and text attributes has not been explored yet, though text and pose attributes independently have been proven to be effective in numerous computer vision tasks. In this paper, we present the first pose augmented Vision-language model (VLM) for VAR. Notably, our scheme achieves an accuracy of 92.81% and 73.02% on two popular human video action recognition benchmark datasets, UCF-101 and HMDB-51, respectively, even without any video data pre-training, and an accuracy of 96.11% and 75.75% after kinetics pre-training.

View on arXivView PDF

Code Implementations(1)

Soumyabrata2003/ViLPOfficial100%
MIT

This is the official implementation of ViLP (ICVGIP'23 Oral), which leverages cross-modal bridge to enhance video recognition by exploring tri-directional knowledge.

50Shell, PythonAug 12, 20232 years agoMIT

Similar Papers

KRAST: Knowledge-Augmented Robotic Action Recognition with Structured Text for Vision-Language Models

86% match2025

SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living

85% match1 repo262025

ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognition

84% match2024

Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition

84% match1 repo492024

Grounding Foundational Vision Models with 3D Human Poses for Robust Action Recognition

84% match2025

Paxion: Patching Action Knowledge in Video-Language Foundation Models

83% match1 repo382026

Cite this paper

@article{chaudhuri2023vilp,
  title  = {ViLP: Knowledge Exploration using Vision, Language, and Pose Embeddings for Video Action Recognition},
  author = {Soumyabrata Chaudhuri and Saumik Bhattacharya},
  year   = {2023},
  eprint = {2308.03908},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.03908v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.03908.svg)](https://codeofpaper.com/paper/2308.03908)

Discussion