Attending to Mathematical Language with Transformers

Artit Wangperawong
12/5/2018
cs.CLcs.LG
transformer-attention

Official code: tensorflow/tensor2tensor (17,475 stars, Apache-2.0, last commit 2023-06-02).

Abstract

Mathematical expressions were generated, evaluated and used to train neural network models based on the transformer architecture. The expressions and their targets were analyzed as a character-level sequence transduction task in which the encoder and decoder are built on attention mechanisms. Three models were trained to understand and evaluate symbolic variables and expressions in mathematics: (1) the self-attentive and feed-forward transformer without recurrence or convolution, (2) the universal transformer with recurrence, and (3) the adaptive universal transformer with recurrence and adaptive computation time. The models respectively achieved test accuracies as high as 76.1%, 78.8% and 84.9% in evaluating the expressions to match the target values. For the cases inferred incorrectly, the results differed from the targets by only one or two characters. The models notably learned to add, subtract and multiply both positive and negative decimal numbers of variable digits assigned to symbolic variables.

View on arXivView PDF

Code Implementations(1)

tensorflow/tensor2tensorOfficial100%
Apache-2.0

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.

17,4753,800C++, HTMLJun 15, 20173 years agoApache-2.0
deep-learningmachine-learningmachine-translationreinforcement-learningtpu

Similar Papers

ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modelin

84% match2026

Fully Interpretable Minimal Transformers: From Geometry to Algorithm

83% match2026

On Limitation of Transformer for Learning HMMs

83% match2024

AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures

82% match1 repo12024

Thinking Like Transformers

82% match1 repo3452021

The Progression of Transformers from Language to Vision to MOT: A Literature Review on Multi-Object Tracking with Transformers

82% match2024

Cite this paper

@article{wangperawong2018attending,
  title  = {Attending to Mathematical Language with Transformers},
  author = {Artit Wangperawong},
  year   = {2018},
  eprint = {1812.02825},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1812.02825}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1812.02825.svg)](https://codeofpaper.com/paper/1812.02825)

Discussion