BrainTransformers: SNN-LLM

Zhengzheng Tang, Eva Zhu
10/3/2024
cs.NEcs.CLcs.LG

No official code found. Best implementation: LumenScopeAI/BrainTransformers-SNN-LLM (36 stars, last commit 2024-10-22).

Abstract

This study introduces BrainTransformers, an innovative Large Language Model (LLM) implemented using Spiking Neural Networks (SNN). Our key contributions include: (1) designing SNN-compatible Transformer components such as SNNMatmul, SNNSoftmax, and SNNSiLU; (2) implementing an SNN approximation of the SiLU activation function; and (3) developing a Synapsis module to simulate synaptic plasticity. Our 3-billion parameter model, BrainTransformers-3B-Chat, demonstrates competitive performance across various benchmarks, including MMLU (63.2), BBH (54.1), ARC-C (54.3), and GSM8K (76.3), while potentially offering improved energy efficiency and biological plausibility. The model employs a three-stage training approach, including SNN-specific neuronal synaptic plasticity training. This research opens new avenues for brain-like AI systems in natural language processing and neuromorphic computing. Future work will focus on hardware optimization, developing specialized SNN fine-tuning tools, and exploring practical applications in energy-efficient computing environments.

View on arXivView PDF

Code Implementations(1)

LumenScopeAI/BrainTransformers-SNN-LLMCommunity58%

Based on BrainTransformers, BrainGPTForCausalLM is a Large Language Model (LLM) implemented using Spiking Neural Networks (SNN). We are excited to announce that our technical report is now available on arXiv: [BrainTransformers: SNN-LLM](https://arxiv.org/abs/2410.14687)

363C, C++Sep 30, 20241 years ago

Similar Papers

Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model

87% match1 repo102024

Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directions

87% match2024

LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models

86% match1 repo72025

SpikingBrain: Spiking Brain-inspired Large Models

85% match2025

SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking

85% match1 repo582024

FAS: Fast ANN-SNN Conversion for Spiking Large Language Models

85% match1 repo32025

Cite this paper

@article{tang2024braintransformers,
  title  = {BrainTransformers: SNN-LLM},
  author = {Zhengzheng Tang and Eva Zhu},
  year   = {2024},
  eprint = {2410.14687},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.14687v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.14687.svg)](https://codeofpaper.com/paper/2410.14687)

Discussion