From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces

Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat +4 more
5/31/2023
cs.LGcs.CLcs.CV

No official code found. Best implementation: google-deepmind/pix2act (60 stars, Apache-2.0, last commit 2024-01-09).

Abstract

Much of the previous work towards digital agents for graphical user interfaces (GUIs) has relied on text-based representations (derived from HTML or other structured data sources), which are not always readily available. These input representations have been often coupled with custom, task-specific action spaces. This paper focuses on creating agents that interact with the digital world using the same conceptual interface that humans commonly use -- via pixel-based screenshots and a generic action space corresponding to keyboard and mouse actions. Building upon recent progress in pixel-based pretraining, we show, for the first time, that it is possible for such agents to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based instruction following tasks.

View on arXivView PDF

Code Implementations(1)

google-deepmind/pix2actCommunity100%
Apache-2.0
608PythonNov 13, 20232 years agoApache-2.0

Similar Papers

ShowUI-Aloha: Human-Taught GUI Agent

83% match2026

Instruction Agent: Enhancing Agent with Expert Demonstration

83% match2025

UI-TARS: Pioneering Automated GUI Interaction with Native Agents

83% match1 repo11.6k2025

ShowUI: One Vision-Language-Action Model for GUI Visual Agent

82% match1 repo1.9k2026

Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

81% match1 repo3162026

SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis

80% match2026

Cite this paper

@article{shaw2023from,
  title  = {From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces},
  author = {Peter Shaw and Mandar Joshi and James Cohan and Jonathan Berant and Panupong Pasupat and Hexiang Hu and Urvashi Khandelwal and Kenton Lee and Kristina Toutanova},
  year   = {2023},
  eprint = {2306.00245},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.00245v2},
  journal = {NEURIPS 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.00245.svg)](https://codeofpaper.com/paper/2306.00245)

Discussion