Reliable, Reproducible, and Really Fast Leaderboards with Evalica

Dmitry Ustalov
12/15/2024
cs.CL

No official code found. Best implementation: dustalov/evalica (68 stars, Apache-2.0, last commit 2026-04-29).

Abstract

The rapid advancement of natural language processing (NLP) technologies, such as instruction-tuned large language models (LLMs), urges the development of modern evaluation protocols with human and machine feedback. We introduce Evalica, an open-source toolkit that facilitates the creation of reliable and reproducible model leaderboards. This paper presents its design, evaluates its performance, and demonstrates its usability through its Web interface, command-line interface, and Python API.

View on arXivView PDF

Code Implementations(1)

dustalov/evalicaCommunity100%
Apache-2.0

Evalica, your favourite evaluation toolkit

685Rust, PythonJun 15, 20245 months agoApache-2.0
arenabradley-terryeloevalicaevals+15 more

Similar Papers

Prompt-to-Leaderboard

84% match1 repo2752025

Evalverse: Unified and Accessible Library for Large Language Model Evaluation

83% match1 repo2352024

League: Leaderboard Generation on Demand

82% match2025

LEGOBench: Scientific Leaderboard Generation Benchmark

81% match1 repo42024

Improving LLM Leaderboards with Psychometrical Methodology

81% match2025

CLEVA: Chinese Language Models EVAluation Platform

80% match1 repo642023

Cite this paper

@article{ustalov2024reliable,
  title  = {Reliable, Reproducible, and Really Fast Leaderboards with Evalica},
  author = {Dmitry Ustalov},
  year   = {2024},
  eprint = {2412.11314},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.11314v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.11314.svg)](https://codeofpaper.com/paper/2412.11314)

Discussion