MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents

Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang +23 more
7/25/2025
cs.CVcs.CL

Official code: open-compass/MMBench-GUI (113 stars, last commit 2025-09-08).

Abstract

We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, macOS, Linux, iOS, Android, and Web platforms. It comprises four levels: GUI Content Understanding, Element Grounding, Task Automation, and Task Collaboration, covering essential skills for GUI agents. In addition, we propose a novel Efficiency-Quality Area (EQA) metric to assess GUI agent execution efficiency in online automation scenarios. Through MMBench-GUI, we identify accurate visual grounding as a critical determinant of overall task success, emphasizing the substantial benefits of modular frameworks that integrate specialized grounding modules. Furthermore, to achieve reliable GUI automation, an agent requires strong task planning and cross-platform generalization abilities, with long-context memory, a broad action space, and long-term reasoning playing a critical role. More important, task efficiency remains a critically underexplored dimension, and all models suffer from substantial inefficiencies, with excessive redundant steps even when tasks are ultimately completed. The integration of precise localization, effective planning, and early stopping strategies is indispensable to enable truly efficient and scalable GUI automation. Our benchmark code, evaluation data, and running environment will be publicly available at https://github.com/open-compass/MMBench-GUI.

View on arXivView PDF

Code Implementations(1)

open-compass/MMBench-GUIOfficial100%

Official repo of "MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents". It can be used to evaluate a GUI agent with a hierarchical manner across multiple platforms, including Windows, Linux, macOS, iOS, Android and Web.

1135PythonJun 11, 20251 years ago
benchmark-frameworkcomputer-usegui-agentvision-language-model

Similar Papers

MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment

86% match2026

A Survey on (M)LLM-Based GUI Agents

83% match2025

ColorBench: Benchmarking Mobile Agents with Graph-Structured Framework for Complex Long-Horizon Tasks

82% match1 repo152025

UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction

82% match2026

Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

81% match1 repo3162026

Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents

81% match1 repo9.2k2026

Cite this paper

@article{wang2025mmbenchgui,
  title  = {MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents},
  author = {Xuehui Wang and Zhenyu Wu and JingJing Xie and Zichen Ding and Bowen Yang and Zehao Li and Zhaoyang Liu and Qingyun Li and Xuan Dong and Zhe Chen and Weiyun Wang and Xiangyu Zhao and Jixuan Chen and Haodong Duan and Tianbao Xie and Chenyu Yang and Shiqian Su and Yue Yu and Yuan Huang and Yiqian Liu and Xiao Zhang and Yanting Zhang and Xiangyu Yue and Weijie Su and Xizhou Zhu and Wei Shen and Jifeng Dai and Wenhai Wang},
  year   = {2025},
  eprint = {2507.19478},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2507.19478v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2507.19478.svg)](https://codeofpaper.com/paper/2507.19478)

Discussion