The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang +5 more
2/27/2024
cs.CLcs.LG

No official code found. Best implementation: rejunity/tiny-asic-1_58bit-matrix-mul (194 stars, Apache-2.0, last commit 2024-04-19). 1 more below.

Abstract

Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption. More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for training new generations of LLMs that are both high-performance and cost-effective. Furthermore, it enables a new computation paradigm and opens the door for designing specific hardware optimized for 1-bit LLMs.

View on arXivView PDF

Code Implementations(2)

rejunity/tiny-asic-1_58bit-matrix-mulCommunity67%
Apache-2.0

Tiny ASIC implementation for "The Era of 1-bit LLMs All Large Language Models are in 1.58 Bits" matrix multiplication unit

19415Tcl, PythonMar 5, 20242 years agoApache-2.0
asicllm-inferencesystolic-arraystinytapeout
dhakalnirajan/LLaMA-BitNetCommunity59%
MIT

LLaMA-BitNet is a repository dedicated to empowering users to train their own BitNet models built upon LLaMA 2 model, inspired by the groundbreaking paper 'The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits'.

374PythonMar 30, 20247 months agoMIT
large-language-modelsllamallama2llmllms+2 more

Similar Papers

BitNet b1.58 2B4T Technical Report

91% match2025

BitNet a4.8: 4-bit Activations for 1-bit LLMs

87% match2024

Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models

87% match2025

Matmul or No Matmul in the Era of 1-bit LLMs

87% match2024

1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs

86% match1 repo39.8k2024

PIM-LLM: A High-Throughput Hybrid PIM Architecture for 1-bit LLMs

86% match2025

Cite this paper

@article{ma2024bit,
  title  = {The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits},
  author = {Shuming Ma and Hongyu Wang and Lingxiao Ma and Lei Wang and Wenhui Wang and Shaohan Huang and Li Dong and Ruiping Wang and Jilong Xue and Furu Wei},
  year   = {2024},
  eprint = {2402.17764},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.17764v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.17764.svg)](https://codeofpaper.com/paper/2402.17764)

Discussion