A Toolchain for Comprehensive Audio/Video Analysis Using Deep Learning Based Multimodal Approach (A use case of riot or violent context detection)

Lam Pham, Phat Lam, Tin Nguyen, Hieu Tang, Alexander Schindler
5/2/2024
cs.SDcs.AIeess.AS

No confident code implementation found for this paper.

Abstract

In this paper, we present a toolchain for a comprehensive audio/video analysis by leveraging deep learning based multimodal approach. To this end, different specific tasks of Speech to Text (S2T), Acoustic Scene Classification (ASC), Acoustic Event Detection (AED), Visual Object Detection (VOD), Image Captioning (IC), and Video Captioning (VC) are conducted and integrated into the toolchain. By combining individual tasks and analyzing both audio \& visual data extracted from input video, the toolchain offers various audio/video-based applications: Two general applications of audio/video clustering, comprehensive audio/video summary and a specific application of riot or violent context detection. Furthermore, the toolchain presents a flexible and adaptable architecture that is effective to integrate new models for further audio/video-based applications.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Enhancing Human Action Recognition and Violence Detection Through Deep Learning Audiovisual Fusion

84% match2024

Intelligent Image Sensing for Crime Analysis: A ML Approach towards Enhanced Violence Detection and Investigation

82% match2025

Application of Attention Mechanism with Bidirectional Long Short-Term Memory (BiLSTM) and CNN for Human Conflict Detection using Computer Vision

81% match2025

A Multimodal Framework for Deepfake Detection

80% match2024

Comparative Analysis of Image, Video, and Audio Classifiers for Automated News Video Segmentation

80% match2025

Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification

79% match2024

Cite this paper

@article{pham2024toolchain,
  title  = {A Toolchain for Comprehensive Audio/Video Analysis Using Deep Learning Based Multimodal Approach (A use case of riot or violent context detection)},
  author = {Lam Pham and Phat Lam and Tin Nguyen and Hieu Tang and Alexander Schindler},
  year   = {2024},
  eprint = {2407.03110},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.03110v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.03110.svg)](https://codeofpaper.com/paper/2407.03110)

Discussion