Fast Feedforward Networks

Peter Belcak, Roger Wattenhofer
8/28/2023
cs.LGcs.AIcs.PF

No official code found. Best implementation: pbelcak/fastfeedforward (223 stars, MIT, last commit 2024-04-09).

Abstract

We break the linear link between the layer size and its inference cost by introducing the fast feedforward (FFF) architecture, a log-time alternative to feedforward networks. We demonstrate that FFFs are up to 220x faster than feedforward networks, up to 6x faster than mixture-of-experts networks, and exhibit better training properties than mixtures of experts thanks to noiseless conditional execution. Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance.

View on arXivView PDF

Code Implementations(1)

pbelcak/fastfeedforwardCommunity100%
MIT

A repository for log-time feedforward networks

22321Python, Jupyter NotebookAug 27, 20232 years agoMIT

Similar Papers

RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers

83% match1 repo182025

Unrolling Ternary Neural Networks

82% match1 repo112019

DISCO: Distributed Inference with Sparse Communications

80% match2023

Approximating Two-Layer Feedforward Networks for Efficient Transformers

80% match1 repo212023

Feed-Forward Optimization With Delayed Feedback for Neural Network Training

80% match1 repo82023

Communication-Efficient Multi-Device Inference Acceleration for Transformer Models

80% match1 repo32025

Cite this paper

@article{belcak2023fast,
  title  = {Fast Feedforward Networks},
  author = {Peter Belcak and Roger Wattenhofer},
  year   = {2023},
  eprint = {2308.14711},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.14711v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.14711.svg)](https://codeofpaper.com/paper/2308.14711)

Discussion