Generalized Attention Mechanism and Relative Position for Transformer

R. V. R. Pandya
7/24/2022
cs.CLcs.LG

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

In this paper, we propose generalized attention mechanism (GAM) by first suggesting a new interpretation for self-attention mechanism of Vaswani et al. . Following the interpretation, we provide description for different variants of attention mechanism which together form GAM. Further, we propose a new relative position representation within the framework of GAM. This representation can be easily utilized for cases in which elements next to each other in input sequence can be at random locations in actual dataset/corpus.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

How Local Mixing Encodes Relative Position in Global NoPE Attention

84% match2026

Self-Attention with Relative Position Representations

84% match1 repo17.5k2018

Learning Positional Attention for Sequential Recommendation

83% match1 repo32024

Tucker Attention: A generalization of approximate attention mechanisms

83% match2026

Self-Attention as Distributional Projection: A Unified Interpretation of Transformer Architecture

83% match2025

Mapping of attention mechanisms to a generalized Potts model

83% match2023

Cite this paper

@article{pandya2022generalized,
  title  = {Generalized Attention Mechanism and Relative Position for Transformer},
  author = {R. V. R. Pandya},
  year   = {2022},
  eprint = {2208.10247},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2208.10247v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2208.10247.svg)](https://codeofpaper.com/paper/2208.10247)

Discussion