RWKV-UI: UI Understanding with Enhanced Perception and Reasoning

Jiaxi Yang, Haowen Hou
2/6/2025
cs.CVcs.HC

No confident code implementation found for this paper.

Abstract

Existing Visual Language Modelsoften struggle with information loss and limited reasoning abilities when handling high-resolution web interfaces that combine complex visual, textual, and interactive elements. These challenges are particularly evident in tasks requiring webpage layout comprehension and multi-step interactive reasoning. To address these challenges, we propose RWKV-UI, a Visual Language Model based on the RWKV architecture, specifically designed to handle high-resolution UI images. During model training, we introduce layout detection as a visual prompt to help the model better understand the webpage layout structures. Additionally, we design a visual prompt based on the Chain-of-Thought(CoT) mechanism, which enhances the model's ability to understand and reason about webpage content through reasoning chains. Experimental results show that RWKV-UI demonstrates significant performance improvements in high-resolution UI understanding and interactive reasoning tasks.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding

84% match1 repo6932022

Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding

83% match1 repo62026

PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understanding

83% match2025

Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension

83% match2026

Efficient End-to-End Visual Document Understanding with Rationale Distillation

83% match2023

Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach

83% match2026

Cite this paper

@article{yang2025rwkvui,
  title  = {RWKV-UI: UI Understanding with Enhanced Perception and Reasoning},
  author = {Jiaxi Yang and Haowen Hou},
  year   = {2025},
  eprint = {2502.03971},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2502.03971v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2502.03971.svg)](https://codeofpaper.com/paper/2502.03971)

Discussion