TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models

Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu +4 more
9/21/2021
cs.CLcs.CV

Official code: microsoft/unilm (22,168 stars, MIT, last commit 2026-01-23).

Abstract

Text recognition is a long-standing research problem for document digitalization. Existing approaches are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on the printed, handwritten and scene text recognition tasks. The TrOCR models and code are publicly available at \url{https://aka.ms/trocr}.

View on arXivView PDF

Code Implementations(1)

microsoft/unilmOfficial100%
MIT

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

22,1682,703C, C++Jul 23, 20199 months agoMIT
beitbeit-3bitnetdeepnetdocument-ai+15 more

Similar Papers

LEGO: Self-Supervised Representation Learning for Scene Text Images

85% match2024

TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models

85% match2024

Turning a CLIP Model into a Scene Text Detector

84% match1 repo2002023

A Context-Driven Training-Free Network for Lightweight Scene Text Segmentation and Recognition

84% match2025

MonkeyOCRv2: A Visual-Text Foundation Model for Document AI

84% match2026

Text Image Generation for Low-Resource Languages with Dual Translation Learning

84% match2024

Cite this paper

@article{li2021trocr,
  title  = {TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models},
  author = {Minghao Li and Tengchao Lv and Jingye Chen and Lei Cui and Yijuan Lu and Dinei Florencio and Cha Zhang and Zhoujun Li and Furu Wei},
  year   = {2021},
  eprint = {2109.10282},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2109.10282}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2109.10282.svg)](https://codeofpaper.com/paper/2109.10282)

Discussion