PolySmart @ TRECVid 2024 Medical Video Question Answering

Jiaxin Wu, Yiyang Jiang, Xiao-Yong Wei, Qing Li
12/20/2024
cs.CVcs.MM

No confident code implementation found for this paper.

Abstract

Video Corpus Visual Answer Localization (VCVAL) includes question-related video retrieval and visual answer localization in the videos. Specifically, we use text-to-text retrieval to find relevant videos for a medical question based on the similarity of video transcript and answers generated by GPT4. For the visual answer localization, the start and end timestamps of the answer are predicted by the alignments on both visual content and subtitles with queries. For the Query-Focused Instructional Step Captioning (QFISC) task, the step captions are generated by GPT4. Specifically, we provide the video captions generated by the LLaVA-Next-Video model and the video subtitles with timestamps as context, and ask GPT4 to generate step captions for the given medical query. We only submit one run for evaluation and it obtains a F-score of 11.92 and mean IoU of 9.6527.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Towards Answering Health-related Questions from Medical Videos: Datasets and Approaches

84% match2023

ObjectNLQ @ Ego4D Episodic Memory Challenge 2024

83% match1 repo12024

Overview of the NLPCC 2026 Shared Task 1: Difficulty-Aware Multilingual and Multimodal Medical Instructional Video Understanding Evaluation

82% match2026

Temporal Perceiving Video-Language Pre-training

81% match2023

MINOTAUR: Multi-task Video Grounding From Multimodal Queries

81% match1 repo2023

Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins

80% match2026

Cite this paper

@article{wu2024polysmart,
  title  = {PolySmart @ TRECVid 2024 Medical Video Question Answering},
  author = {Jiaxin Wu and Yiyang Jiang and Xiao-Yong Wei and Qing Li},
  year   = {2024},
  eprint = {2412.15514},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.15514v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.15514.svg)](https://codeofpaper.com/paper/2412.15514)

Discussion