A Transformer with Stack Attention

Jiaoda Li, Jennifer C. White, Mrinmaya Sachan, Ryan Cotterell
5/7/2024
cs.CL

No official code found. Best implementation: rycolab/stack-transformer (4 stars, last commit 2024-05-20).

Abstract

Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free language tasks. In an attempt to address this limitation in the modeling power of transformer-based language models, we propose augmenting them with a differentiable, stack-based attention mechanism. Our stack-based attention mechanism can be incorporated into any transformer-based language model and adds a level of interpretability to the model. We show that the addition of our stack-based attention mechanism enables the transformer to model some, but not all, deterministic context-free languages.

View on arXivView PDF

Code Implementations(1)

rycolab/stack-transformerCommunity100%
40PythonJul 28, 20232 years ago

Similar Papers

Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns

87% match1 repo192026

Selective Attention Improves Transformer

86% match2024

Deconstructing Attention: Investigating Design Principles for Effective Language Modeling

85% match2025

Attention-likelihood relationship in transformers

85% match1 repo42023

KV Shifting Attention Enhances Language Modeling

85% match1 repo1172024

SAP: Syntactic Attention Pruning for Transformer-based Language Models

85% match2025

Cite this paper

@article{li2024transformer,
  title  = {A Transformer with Stack Attention},
  author = {Jiaoda Li and Jennifer C. White and Mrinmaya Sachan and Ryan Cotterell},
  year   = {2024},
  eprint = {2405.04515},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.04515v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.04515.svg)](https://codeofpaper.com/paper/2405.04515)

Discussion