Image-to-Markup Generation with Coarse-to-Fine Attention

Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, Alexander M. Rush
9/16/2016
cs.CVcs.CLcs.LG
transformer-attention

Official code: harvardnlp/im2markup (1,256 stars, MIT, last commit 2026-10-19).

Abstract

We present a neural encoder-decoder model to convert images into presentational markup based on a scalable coarse-to-fine attention mechanism. Our method is evaluated in the context of image-to-LaTeX generation, and we introduce a new dataset of real-world rendered mathematical expressions paired with LaTeX markup. We show that unlike neural OCR techniques using CTC-based models, attention-based approaches can tackle this non-standard OCR task. Our approach outperforms classical mathematical OCR systems by a large margin on in-domain rendered data, and, with pretraining, also performs well on out-of-domain handwritten data. To reduce the inference complexity associated with the attention-based approaches, we introduce a new coarse-to-fine attention layer that selects a support region before applying attention.

View on arXivView PDF

Code Implementations(1)

harvardnlp/im2markupOfficial100%
MIT

Neural model for converting Image-to-Markup (by Yuntian Deng yuntiandeng.com)

1,256214Lua, TeXSep 16, 20165 days agoMIT

Similar Papers

Self-Supervised Learning for Text Recognition: A Critical Survey

79% match2024

A Hybrid Vision Transformer Approach for Mathematical Expression Recognition

78% match2026

Decoupled Attention Network for Text Recognition

77% match2 repos4922019

FastOCR: Dynamic Visual Fixation via KV Cache Pruning for Efficient Document Parsing

77% match2026

When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition

77% match1 repo3872022

Evaluating Synthetic Pre-Training for Handwriting Processing Tasks

77% match1 repo52023

Cite this paper

@article{deng2016imagetomarkup,
  title  = {Image-to-Markup Generation with Coarse-to-Fine Attention},
  author = {Yuntian Deng and Anssi Kanervisto and Jeffrey Ling and Alexander M. Rush},
  year   = {2016},
  eprint = {1609.04938},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1609.04938}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1609.04938.svg)](https://codeofpaper.com/paper/1609.04938)

Discussion