Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber +1 more
10/15/2019
cs.LGstat.ML

Official code: ischlag/TP-Transformer (49 stars, MIT, last commit 2021-03-08).

Abstract

We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our Tensor-Product Transformer (TP-Transformer) sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-problems. The essential component of the model is a novel attention mechanism, called TP-Attention, which explicitly encodes the relations between each Transformer cell and the other cells from which values have been retrieved by attention. TP-Attention goes beyond linear combination of retrieved values, strengthening representation-building and resolving ambiguities introduced by multiple layers of standard attention. The TP-Transformer's attention maps give better insights into how it is capable of solving the Mathematics Dataset's challenging problems. Pretrained models and code will be made available after publication.

View on arXivView PDF

Code Implementations(1)

ischlag/TP-TransformerOfficial100%
MIT
497Shell, PythonOct 28, 20195 years agoMIT

Similar Papers

Rethinking Self-Attention: Towards Interpretability in Neural Parsing

80% match1 repo1452019

RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling

80% match2023

Knowledge Guided Bidirectional Attention Network for Human-Object Interaction Detection

80% match2022

Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models

80% match1 repo742023

Enhancing LLM Intelligence with ARM-RAG: Auxiliary Rationale Memory for Retrieval Augmented Generation

79% match2023

HGNet: Scalable Foundation Model for Automated Knowledge Graph Generation from Scientific Literature

78% match1 repo22026

Cite this paper

@article{schlag2019enhancing,
  title  = {Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving},
  author = {Imanol Schlag and Paul Smolensky and Roland Fernandez and Nebojsa Jojic and Jürgen Schmidhuber and Jianfeng Gao},
  year   = {2019},
  eprint = {1910.06611},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1910.06611}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1910.06611.svg)](https://codeofpaper.com/paper/1910.06611)

Discussion