Insights from Benchmarking Frontier Language Models on Web App Code Generation

Yi Cui
9/8/2024
cs.SEcs.AI

No official code found. Best implementation: onekq/WebApp1k (16 stars, MIT, last commit 2025-11-21).

Abstract

This paper presents insights from evaluating 16 frontier large language models (LLMs) on the WebApp1K benchmark, a test suite designed to assess the ability of LLMs to generate web application code. The results reveal that while all models possess similar underlying knowledge, their performance is differentiated by the frequency of mistakes they make. By analyzing lines of code (LOC) and failure distributions, we find that writing correct code is more complex than generating incorrect code. Furthermore, prompt engineering shows limited efficacy in reducing errors beyond specific cases. These findings suggest that further advancements in coding LLM should emphasize on model reliability and mistake minimization.

View on arXivView PDF

Code Implementations(1)

onekq/WebApp1kCommunity100%
MIT

WebApp1k benchmark

160Python, JavaScriptJul 21, 202411 months agoMIT

Similar Papers

Holistic Evaluation of State-of-the-Art LLMs for Code Generation

90% match2025

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason

88% match2025

Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation

88% match2025

Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models

87% match1 repo142024

Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specifications

87% match2025

Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks

86% match2025

Cite this paper

@article{cui2024insights,
  title  = {Insights from Benchmarking Frontier Language Models on Web App Code Generation},
  author = {Yi Cui},
  year   = {2024},
  eprint = {2409.05177},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.05177v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.05177.svg)](https://codeofpaper.com/paper/2409.05177)

Discussion