Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers

Xiuying Wei, Skander Moalla, Razvan Pascanu, Caglar Gulcehre
2/3/2026

No official code found. Best implementation: CLAIRE-Labo/StructuredFFN (19 stars, last commit 2024-07-24).

Semantic Scholar

Code Implementations(1)

CLAIRE-Labo/StructuredFFNCommunity70%

The official code of "Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers"

193Shell, PythonJun 23, 20242 years ago

Cite this paper

@article{wei2026building,
  title  = {Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers},
  author = {Xiuying Wei and Skander Moalla and Razvan Pascanu and Caglar Gulcehre},
  year   = {2026},
  url    = {https://www.semanticscholar.org/paper/e4e52c4a5611d10c17115e5f302cbb122ecafdee},
  journal = {NEURIPS 2024 2024}
}

Discussion