An All-MLP Sequence Modeling Architecture That Excels at Copying

Chenwei Cui, Zehao Yan, Gedeon Muhawenayo, Hannah Kerner
6/23/2024
cs.LG

No official code found. Best implementation: kerner-lab/causal-relation-networks (5 stars, Apache-2.0, last commit 2024-07-20). 1 more below.

Abstract

Recent work demonstrated Transformers' ability to efficiently copy strings of exponential sizes, distinguishing them from other architectures. We present the Causal Relation Network (CausalRN), an all-MLP sequence modeling architecture that can match Transformers on the copying task. Extending Relation Networks (RNs), we implemented key innovations to support autoregressive sequence modeling while maintaining computational feasibility. We discovered that exponentially-activated RNs are reducible to linear time complexity, and pre-activation normalization induces an infinitely growing memory pool, similar to a KV cache. In ablation study, we found both exponential activation and pre-activation normalization are indispensable for Transformer-level copying. Our findings provide new insights into what actually constitutes strong in-context retrieval.

View on arXivView PDF

Code Implementations(2)

kerner-lab/causal-relation-networksCommunity100%
Apache-2.0

An All-MLP Sequence Modeling Architecture That Excels at Copying (Casual Relational Networks) Resources. https://arxiv.org/abs/2406.16168

50PythonJul 20, 20242 years agoApache-2.0
FrankCuiCN/Causal-Relation-NetworksCommunity69%
MIT

An All-MLP Sequence Modeling Architecture That Excels at Copying (Casual Relational Networks)

21PythonJun 21, 20241 years agoMIT

Similar Papers

RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval

82% match1 repo272024

RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval

82% match1 repo272026

Scaling Transformer to 1M tokens and beyond with RMT

81% match2 repos7762023

Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling

81% match1 repo152025

Repeat After Me: Transformers are Better than State Space Models at Copying

81% match1 repo402024

A Unified View of Long-Sequence Models towards Modeling Million-Scale Dependencies

80% match2023

Cite this paper

@article{cui2024allmlp,
  title  = {An All-MLP Sequence Modeling Architecture That Excels at Copying},
  author = {Chenwei Cui and Zehao Yan and Gedeon Muhawenayo and Hannah Kerner},
  year   = {2024},
  eprint = {2406.16168},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.16168v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.16168.svg)](https://codeofpaper.com/paper/2406.16168)

Discussion