Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru +12 more
2/29/2024
cs.LGcs.CL

No official code found. Best implementation: kyegomez/Griffin (58 stars, MIT, PyTorch, last commit 2025-10-27). 1 more below.

Abstract

Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale. We propose Hawk, an RNN with gated linear recurrences, and Griffin, a hybrid model that mixes gated linear recurrences with local attention. Hawk exceeds the reported performance of Mamba on downstream tasks, while Griffin matches the performance of Llama-2 despite being trained on over 6 times fewer tokens. We also show that Griffin can extrapolate on sequences significantly longer than those seen during training. Our models match the hardware efficiency of Transformers during training, and during inference they have lower latency and significantly higher throughput. We scale Griffin up to 14B parameters, and explain how to shard our models for efficient distributed training.

View on arXivView PDF

Code Implementations(2)

kyegomez/GriffinCommunity74%
PyTorchMIT

Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"

585Shell, PythonMar 4, 202411 months agoMIT
aifeedforwardmlmlpneural-network+4 more
fattorib/hawk-pytorchCommunity62%
Apache-2.0

PyTorch implementation of Hawk from "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models" (https://arxiv.org/abs/2402.19427). Compatible with torch.compile.

60PythonSep 2, 20241 years agoApache-2.0

Cite this paper

@article{de2024griffin,
  title  = {Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models},
  author = {Soham De and Samuel L. Smith and Anushan Fernando and Aleksandar Botev and George Cristian-Muraru and Albert Gu and Ruba Haroun and Leonard Berrada and Yutian Chen and Srivatsan Srinivasan and Guillaume Desjardins and Arnaud Doucet and David Budden and Yee Whye Teh and Razvan Pascanu and Nando De Freitas and Caglar Gulcehre},
  year   = {2024},
  eprint = {2402.19427},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.19427v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.19427.svg)](https://codeofpaper.com/paper/2402.19427)

Discussion