Multimodal deep networks for text and image-based document classification

Nicolas Audebert, Catherine Herold, Kuider Slimani, Cédric Vidal
7/15/2019
cs.CV

Official code: QuickSign/ocrized-text-dataset (45 stars, Apache-2.0, last commit 2019-05-07).

Abstract

Classification of document images is a critical step for archival of old manuscripts, online subscription and administrative procedures. Computer vision and deep learning have been suggested as a first solution to classify documents based on their visual appearance. However, achieving the fine-grained classification that is required in real-world setting cannot be achieved by visual analysis alone. Often, the relevant information is in the actual text content of the document. We design a multimodal neural network that is able to learn from word embeddings, computed on text extracted by OCR, and from the image. We show that this approach boosts pure image accuracy by 3% on Tobacco3482 and RVL-CDIP augmented by our new QS-OCR text dataset (https://github.com/Quicksign/ocrized-text-dataset), even without clean text information.

View on arXivView PDF

Code Implementations(1)

QuickSign/ocrized-text-datasetOfficial100%
Apache-2.0

Quicksign OCRized Text Dataset (QS-OCR)

457Shell, PythonApr 26, 20197 years agoApache-2.0

Similar Papers

EAML: Ensemble Self-Attention-based Mutual Learning Network for Document Image Classification

83% match2023

Enhancing Document Information Analysis with Multi-Task Pre-training: A Robust Approach for Information Extraction in Visually-Rich Documents

82% match2023

Text Classification: A Perspective of Deep Learning Methods

82% match2023

Comparing Deep Neural Network for Multi-Label ECG Diagnosis From Scanned ECG

82% match2025

Pattern Spotting and Image Retrieval in Historical Documents using Deep Hashing

82% match2022

Task-driven single-image super-resolution reconstruction of document scans

81% match2024

Cite this paper

@article{audebert2019multimodal,
  title  = {Multimodal deep networks for text and image-based document classification},
  author = {Nicolas Audebert and Catherine Herold and Kuider Slimani and Cédric Vidal},
  year   = {2019},
  eprint = {1907.06370},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1907.06370}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1907.06370.svg)](https://codeofpaper.com/paper/1907.06370)

Discussion