WAFFLE: Finetuning Multi-Modal Model for Automated Front-End Development

Shanchao Liang, Nan Jiang, Shangshu Qian, Lin Tan
10/24/2024
cs.SEcs.CLcs.CV

No official code found. Best implementation: lt-asset/Waffle (12 stars, BSD-3-Clause, last commit 2025-05-28).

Abstract

Web development involves turning UI designs into functional webpages, which can be difficult for both beginners and experienced developers due to the complexity of HTML's hierarchical structures and styles. While Large Language Models (LLMs) have shown promise in generating source code, two major challenges persist in UI-to-HTML code generation: (1) effectively representing HTML's hierarchical structure for LLMs, and (2) bridging the gap between the visual nature of UI designs and the text-based format of HTML code. To tackle these challenges, we introduce Waffle, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs' understanding of HTML's structure and a contrastive fine-tuning approach to align LLMs' understanding of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code, outperforming current fine-tuning methods.

View on arXivView PDF

Code Implementations(1)

lt-asset/WaffleCommunity100%
BSD-3-Clause

For ACL25 paper "WAFFLE: Multi-Modal Model for Automated Front-End Development" - by Shanchao Liang and Nan Jiang and Shangshu Qian and Lin Tan

123HTML, PythonOct 23, 20241 years agoBSD-3-Clause
front-end-developmentllmllm4codereverse-engineeringweb-development

Similar Papers

UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback

81% match2024

Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs

80% match1 repo1032026

Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs

80% match1 repo282025

Harnessing Webpage UIs for Text-Rich Visual Understanding

80% match1 repo542026

WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code

79% match1 repo52025

VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?

78% match1 repo682024

Cite this paper

@article{liang2024waffle,
  title  = {WAFFLE: Finetuning Multi-Modal Model for Automated Front-End Development},
  author = {Shanchao Liang and Nan Jiang and Shangshu Qian and Lin Tan},
  year   = {2024},
  eprint = {2410.18362},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.18362v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.18362.svg)](https://codeofpaper.com/paper/2410.18362)

Discussion