Do Current Video LLMs Have Strong OCR Abilities? A Preliminary Study

Yulin Fei, Yuhui Gao, Xingyuan Xian, Xiaojin Zhang, Tao Wu +1 more
12/29/2024
cs.CV

Official code: YuHuiGao/FG-Bench (7 stars, Apache-2.0, last commit 2025-06-08).

Abstract

With the rise of multimodal large language models, accurately extracting and understanding textual information from video content, referred to as video based optical character recognition (Video OCR), has become a crucial capability. This paper introduces a novel benchmark designed to evaluate the video OCR performance of multi-modal models in videos. Comprising 1,028 videos and 2,961 question-answer pairs, this benchmark proposes several key challenges through 6 distinct subtasks: (1) Recognition of text content itself and its basic visual attributes, (2)Semantic and Spatial Comprehension of OCR objects in videos (3) Dynamic Motion detection and Temporal Localization. We developed this benchmark using a semi-automated approach that integrates the OCR ability of image LLMs with manual refinement, balancing efficiency, cost, and data quality. Our resource aims to help advance research in video LLMs and underscores the need for improving OCR ability for video LLMs. The benchmark will be released on https://github.com/YuHuiGao/FG-Bench.git.

View on arXivView PDF

Code Implementations(1)

YuHuiGao/FG-BenchOfficial100%
Apache-2.0

Do Current Video LLMs Have Strong OCR Abilities? A Preliminary Study

70PythonDec 13, 20241 years agoApache-2.0

Similar Papers

MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens

87% match1 repo6402024

VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?

86% match1 repo302024

StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification

86% match1 repo842024

HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data

86% match1 repo7.1k2024

MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding

86% match1 repo4.3k2026

MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeX

85% match2025

Cite this paper

@article{fei2024current,
  title  = {Do Current Video LLMs Have Strong OCR Abilities? A Preliminary Study},
  author = {Yulin Fei and Yuhui Gao and Xingyuan Xian and Xiaojin Zhang and Tao Wu and Wei Chen},
  year   = {2024},
  eprint = {2412.20613},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.20613v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.20613.svg)](https://codeofpaper.com/paper/2412.20613)

Discussion