Breaking the Attention Bottleneck

Kalle Hilsenbek
6/16/2024
cs.LGcs.CL

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Attention-based transformers have become the standard architecture in many deep learning fields, primarily due to their ability to model long-range dependencies and handle variable-length input sequences. However, the attention mechanism with its quadratic complexity is a significant bottleneck in the transformer architecture. This algorithm is only uni-directional in the decoder and converges to a static pattern in over-parametrized decoder-only models. I address this issue by developing a generative function as attention or activation replacement. It still has the auto-regressive character by comparing each token with the previous one. In my test setting with nanoGPT this yields a smaller loss while having a smaller model. The loss further drops by incorporating an average context vector. This concept of attention replacement is distributed under the GNU AGPL v3 license at https://gitlab.com/Bachstelze/causal_generation.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers

87% match2024

Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers

86% match1 repo132023

NoiseFormer -- Noise Diffused Symmetric Attention Transformer

86% match2026

The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures

85% match2025

Transformer Based Linear Attention with Optimized GPU Kernel Implementation

85% match2025

Selective Attention Improves Transformer

85% match2024

Cite this paper

@article{hilsenbek2024breaking,
  title  = {Breaking the Attention Bottleneck},
  author = {Kalle Hilsenbek},
  year   = {2024},
  eprint = {2406.10906},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.10906v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.10906.svg)](https://codeofpaper.com/paper/2406.10906)

Discussion