Llemma: An Open Language Model For Mathematics

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, S. McAleer +4 more
2/12/2026

No official code found. Best implementation: EleutherAI/gpt-neox (7,470 stars, Apache-2.0, last commit 2026-04-13). 2 more below.

Abstract

We present Llemma, a large language model for mathematics. We continue pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web data containing mathematics, and mathematical code, yielding Llemma. On the MATH benchmark Llemma outperforms all known open base models, as well as the unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is capable of tool use and formal theorem proving without any further finetuning. We openly release all artifacts, including 7 billion and 34 billion parameter models, the Proof-Pile-2, and code to replicate our experiments.

View on arXivView PDF

Code Implementations(3)

EleutherAI/gpt-neoxCommunity100%
Apache-2.0

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

7,4701,127C, C++Dec 22, 20206 months agoApache-2.0
deepspeed-librarygpt-3language-modeltransformers
EleutherAI/math-lmCommunity100%
MIT
1,09395Shell, PythonFeb 2, 20232 years agoMIT
wellecks/llmstepCommunity100%
MIT

llmstep: [L]LM proofstep suggestions in Lean 4.

15418Lean, ShellJul 8, 20232 years agoMIT
leanlean4llmtheorem-proving

Similar Papers

RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics

87% match1 repo202025

MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models

87% match1 repo4552026

MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning

86% match1 repo3832026

LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches

86% match2026

IndiMathBench: Autoformalizing Mathematical Reasoning Problems with a Human Touch

85% match1 repo42025

CodeShell Technical Report

85% match2024

Cite this paper

@article{azerbayev2026llemma,
  title  = {Llemma: An Open Language Model For Mathematics},
  author = {Zhangir Azerbayev and Hailey Schoelkopf and Keiran Paster and Marco Dos Santos and S. McAleer and Albert Q. Jiang and Jia Deng and Stella Biderman and S. Welleck},
  year   = {2026},
  eprint = {2310.10631},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2310.10631},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.10631.svg)](https://codeofpaper.com/paper/2310.10631)

Discussion