MASEval: Extending Multi-Agent Evaluation from Models to Systems

Abstract

The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet many existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a Python library that treats the entire agentic system as the unit of analysis. Important design decisions such as harness and context engineering are first-class citizens. MASEval helps practitioners identify the best implementation for their use case and researchers systematically study agentic systems, opening new avenues for principled system design. Through the first systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that, across models of comparable cost and capability, framework choice matters as much as model choice. MASEval is available under the MIT licence at https://github.com/maseval/MASEval.

Download paper here

Code

Documentation

BibTex

@inproceedings{emde2026maseval,
    title = "{MASE}val: Extending Multi-Agent Evaluation from Models to Systems",
    author = "Emde, Cornelius  and
      Rubinstein, Alexander  and
      Goel, Anmol  and
      Heakl, Ahmed  and
      Yun, Sangdoo  and
      Oh, Seong Joon  and
      Gubri, Martin",
    editor = "Durrett, Greg  and
      Jian, Ping",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 3: System Demonstrations)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-demo.34/",
    doi = "10.18653/v1/2026.acl-demo.34",
    pages = "345--356",
    ISBN = "979-8-89176-392-0",
}

Recommended citation:
Download Paper