MASEval: Extending Multi-Agent Evaluation from Models to Systems

Cornelius Emde, Alexander Rubinstein, Anmol Goel, Ahmed Heakl, Sangdoo Yun +2 more
3/9/2026
cs.AIcs.CLcs.LG

Official code: parameterlab/MASEval (37 stars, MIT, last commit 2026-05-02).

Abstract

The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a framework-agnostic library that treats the entire system as the unit of analysis. Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that framework choice matters as much as model choice. MASEval allows researchers to explore all components of agentic systems, opening new avenues for principled system design, and practitioners to identify the best implementation for their use case. MASEval is available under the MIT licence https://github.com/parameterlab/MASEval.

View on arXivView PDF

Code Implementations(1)

parameterlab/MASEvalOfficial100%
MIT

Multi-Agent LLM Evaluation Docs: https://maseval.readthedocs.io/

3712Just, PythonNov 7, 20255 months agoMIT
agentic-aiagentsaievaluationlarge-language-models+2 more

Similar Papers

An Empirical Study of Agent Developer Practices in AI Agent Frameworks

84% match2025

Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

83% match2024

An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc

82% match2026

EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

81% match2026

Agent-as-a-Judge: Evaluate Agents with Agents

81% match1 repo8002026

ARE: Scaling Up Agent Environments and Evaluations

81% match2025

Cite this paper

@article{emde2026maseval,
  title  = {MASEval: Extending Multi-Agent Evaluation from Models to Systems},
  author = {Cornelius Emde and Alexander Rubinstein and Anmol Goel and Ahmed Heakl and Sangdoo Yun and Seong Joon Oh and Martin Gubri},
  year   = {2026},
  eprint = {2603.08835},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2603.08835v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2603.08835.svg)](https://codeofpaper.com/paper/2603.08835)

Discussion