The Massive Legal Embedding Benchmark (MLEB)

Umar Butler, Abdur-Rahman Butler, Adrian Lucas Malec
10/22/2025
cs.CLcs.AIcs.IR

No official code found. Best implementation: isaacus-dev/mleb (39 stars, MIT, last commit 2026-02-24).

Abstract

We present the Massive Legal Embedding Benchmark (MLEB), the largest, most diverse, and most comprehensive open-source benchmark for legal information retrieval to date. MLEB consists of ten expert-annotated datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering). Seven of the datasets in MLEB were newly constructed in order to fill domain and jurisdictional gaps in the open-source legal information retrieval landscape. We document our methodology in building MLEB and creating the new constituent datasets, and release our code, results, and data openly to assist with reproducible evaluations.

View on arXivView PDF

Code Implementations(1)

isaacus-dev/mlebCommunity63%
MIT

The code used to evaluate embedding models on the Massive Legal Embedding Benchmark (MLEB).

396PythonOct 14, 20257 months agoMIT
aiembeddingsisaacuslawmteb

Similar Papers

Scaling Legal AI: Benchmarking Mamba and Transformers for Statutory Classification and Case Law Retrieval

84% match2025

A Reasoning-Focused Legal Retrieval Benchmark

83% match2025

LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain

83% match1 repo2072024

Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents

83% match1 repo22026

Legal RAG Bench: an end-to-end benchmark for legal RAG

82% match2026

Bridging Legal Knowledge and AI: Retrieval-Augmented Generation with Vector Stores, Knowledge Graphs, and Hierarchical Non-negative Matrix Factorization

81% match1 repo262025

Cite this paper

@article{butler2025massive,
  title  = {The Massive Legal Embedding Benchmark (MLEB)},
  author = {Umar Butler and Abdur-Rahman Butler and Adrian Lucas Malec},
  year   = {2025},
  eprint = {2510.19365},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.19365v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.19365.svg)](https://codeofpaper.com/paper/2510.19365)

Discussion