Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström +9 more
12/18/2024
cs.CLcs.AI

No official code found. Best implementation: AnswerDotAI/ModernBERT (1,754 stars, Apache-2.0, last commit 2026-03-01). 1 more below.

Abstract

Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs.

View on arXivView PDF

Code Implementations(2)

AnswerDotAI/ModernBERTCommunity100%
Apache-2.0

Bringing BERT into modernity via both architecture changes and scaling

1,754147Shell, PythonMay 13, 20247 months agoApache-2.0
bertembeddingsllmnlp
OmerNeter/Thinking-Bert-Hierarchical-Reasoning-ModelCommunity58%
Apache-2.0

An Experimental attempt to achieve a thinking small Encoder Model based on https://arxiv.org/pdf/2412.13663 (ModernBert) and https://arxiv.org/abs/2506.21734 (HierarchicalReasoningModel)

51PythonAug 2, 20251 years agoApache-2.0

Similar Papers

ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance

84% match2025

NarrowBERT: Accelerating Masked Language Model Pretraining and Inference

84% match1 repo82023

FNet: Mixing Tokens with Fourier Transforms

83% match1 repo39.0k2021

llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length

82% match2025

RexBERT: Context Specialized Bidirectional Encoders for E-commerce

82% match2026

TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish

82% match2025

Cite this paper

@article{warner2024smarter,
  title  = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author = {Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
  year   = {2024},
  eprint = {2412.13663},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.13663v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.13663.svg)](https://codeofpaper.com/paper/2412.13663)

Discussion