Project  Code  Dataset

1 Introduction↩︎

Large language models (LLMs) have made notable progress in code generation, task planning, and reasoning [1]–[3]. Building on these advances, recent work has demonstrated the feasibility of end-to-end software development from natural-language instructions [4]–[6]. However, existing text-driven paradigms largely rely on carefully designed or highly detailed prompts as input [7], [8], which introduces practical limitations. First, textual descriptions are limited in precisely specifying detailed visual requirements, making it difficult to accurately define page layouts and maintain visual coherence across multiple pages. Second, functional requirements involving cross-page interactions, such as state management and data synchronisation, are difficult to specify precisely and consistently in natural language.

To alleviate these limitations, image-to-webpage generation has emerged as an alternative paradigm. Instead of relying on textual specifications, this approach takes UI designs as input and reconstructs webpage structure and appearance directly from visual signals. This paradigm better aligns with the common designer workflow of turning mockups into functional prototypes. It also addresses the demand from non-technical users who want functional apps directly from screenshots. Several benchmarks have been developed along this line [9]–[12].

However, these benchmarks largely focus on visual fidelity. They measure only how closely the rendered output matches a reference image, not whether the artifact works as an application. A generated interface may appear correct yet remain a behaviorally inert façade. As illustrated in Figure 1, different models might both produce runnable applications with high visual fidelity and basic navigation capabilities. However, while one model merely reconstructs static layouts, the other accurately infers and realizes the underlying dynamic interactions purely from static screenshots.

This gap reflects a deeper distinction in capability. When interaction behavior is supplied explicitly, through textual instructions or demonstration videos [13]–[15], the model performs specification-following: implementing a target that has already been named. When the input is screenshots alone, the model must perform interaction inference: recovering the missing behavior from visual evidence. No existing benchmark measures interaction inference under image-only input.

We introduce UI2App, a benchmark that makes interaction inference measurable. Each task supplies a set of screenshots and asks the model to emit a runnable multi-page artifact. Screenshots arrive without captions, instructions, or interaction descriptions. The model must therefore reconcile structure, palette, and inferred behavior across multiple routes simultaneously rather than translate one canonical view. The released benchmark comprises \(327\) screenshots organized into \(45\) carefully curated state-coherent screenshot sets. Rather than isolated page images, each set spans multiple routes and captures consistent cross-page state, providing the visual evidence required to reconstruct a complete runnable web application. To evaluate the resulting artifacts, we design an end-to-end evaluation chain that scores each along four dimensions: whether the application builds and renders (EXEC
1
@1 / EXEC
3
@3), whether expected routes are reachable (NRS), how faithfully the rendered UI matches the reference (VFS), and whether the interactions implied by the screenshots are realized (IIS).

Of the four, IIS is the core measure and the hardest to design: static screenshots underdetermine behavior and admit multiple valid realizations, since the same visual state may correspond to different correct implementations of interaction logic (e.g., a search box that can be triggered on Enter, button click, or real-time query). To address this challenge, we organize interactions commonly found in web applications into a taxonomy of seven categories and evaluate each with a rubric-based protocol along three complementary axes. Each inferred interaction is assessed against a category-specific rubric that checks whether the expected trigger, state transition, and resulting interface behavior are correctly realized. This taxonomy-driven design enables structured assessment of inferred behaviors without requiring a unique ground-truth execution trace, while providing fine-grained diagnostic signals beyond a single aggregate score.

We evaluate six frontier VLMs on UI2App. Even the strongest reaches an overall IIS of only \(39.3\), indicating that inferring complete interaction behavior from static screenshots remains a frontier challenge. Moreover, visual fidelity does not imply interaction-inference capability: the leader on visual fidelity (VFS) places fourth on IIS, \(5.2\times\) behind the IIS leader. The gap is widest on cross-route state: on \(S3\)-scope interactions (cross-route persistence), three of six models score exactly zero, and even the best reaches only \(21.6\) out of \(100\).

Our contributions are as follows:

• Benchmark. We release UI2App: \(327\) screenshots in \(45\) state-coherent screenshot sets for runnable, multi-route web applications. UI2App is the first benchmark targeting interaction inference rather than specification-following from image-only input.

• End-to-end evaluation protocol and IIS taxonomy. We introduce a four-metric protocol spanning build executability (EXEC
  1
  @1 / EXEC
  3
  @3), navigation reachability (NRS), visual fidelity (VFS), and interaction inference (IIS), with IIS built on the interaction taxonomy.

• Extensive evaluation and analysis. We benchmark six frontier VLMs and a Qwen2.5-VL scaling ladder, establishing the first baselines for image-only interaction inference and showing that visual fidelity does not imply interaction-inference capability, with cross-route state persistence a frontier-wide bottleneck.

2 Related Work↩︎

2.0.0.1 Static Visual-to-Code.

Building on advances in screenshot and UI understanding [16]–[19], vision-language models have made rapid progress on translating UI visuals into frontend code. Early benchmarks established the screenshot-to-code task with real-world webpages and visual similarity scoring [10], [20], then scaled the paradigm with synthetic and large-scale corpora [21], [22]. Subsequent work shifted the output target from HTML to framework-specific components [23]–[28] and extended generation to multi-page sites by pairing screenshots with a structured list of navigation links and resources [29]. The underlying paradigm treats UI generation as a static rendering problem: tasks specify what an interface looks like, not how it behaves, and runtime interaction is left outside evaluation. In contrast, UI2App inherits the screenshot-based input but reframes the task around behavior, asking models to produce a runnable application whose interactions are tested at execution time rather than a visual replica.

2.0.0.2 Interaction Specified by Text.

Once interaction becomes part of the task, the most straightforward way to specify it is through natural language. WebGen-Bench [7] tasks an LLM agent with building a multi-file website codebase from a textual instruction and grades it through automated functional tests. ArtifactsBench [8] extends this paradigm to interactive artifacts spanning web interfaces, visualisations, and mini-games, scored by an LLM judge. An image-augmented variant further pairs multi-screen prototypes with textual descriptions of intended interactions [13]. These benchmarks explicitly specify the interaction and evaluate compliance with that specification. Instead, UI2App provides only static screenshots, leaving the intended behavior to be inferred from visual cues alone.

2.0.0.3 Interaction Demonstrated Visually.

Another line replaces text with additional visual demonstrations. Interaction2Code [30] provides state-pair screenshots and evaluates interaction categories via Selenium-driven probes. IWR-Bench [14] instead supplies user-interaction videos that record state transitions over time, scored on functional and visual fidelity. In these benchmarks, the target interaction trajectory is provided beforehand and serves as a behavioral oracle against which model outputs are evaluated. UI2App, however, provides a single canonical screenshot per route and no target transition, making multi-state inference an intrinsic part of the task rather than information supplied in the input. Table 1 summarises this positioning across the closest prior benchmarks.

4pt

3 UI2App↩︎

3.1 Task Definition↩︎

A UI2App task supplies only a set of \(M\) screenshots (\(M = 4\)–\(14\), mean \(7.3\)), each capturing a distinct route or visual state of the target application. No action trace or interaction description is provided. Given this input, the model must produce the source code of a runnable application that reproduces the screenshots. All tasks and models share a fixed React + TypeScript scaffold, chosen as the most widely deployed front-end stack so that the benchmark reflects current production practice.

3.2 Dataset Construction↩︎

3.2.0.1 Source pool and filtering.

We curate UI2App’s reference screenshots from open-source GitHub projects. To obtain a broad initial pool, we use \(24\) archetype-aware GitHub search queries spanning \(12\) application categories, yielding \(2{,}013\) candidate repositories. We then apply a four-stage automated pipeline that checks permissive licensing, structural validity, buildability, and authentication-wall detection. This leaves \(164\) repositories for expert review.

3.2.0.2 Three-level expert selection.

From the \(164\) usable repositories, expert reviewers select the \(45\) runnable multi-route reference applications using a three-level rubric. The rubric follows a progressive selection logic: page-level discriminability, application-level complexity, and corpus-level diversity. Rather than maximizing corpus size, we prioritize applications whose screenshots contain sufficient visual evidence for route recognition and latent interaction logic inference.

• Page-level discriminability. Each retained screenshot must provide enough visual and semantic evidence for evaluation. Reviewers prefer pages with non-trivial layouts, diverse content regions, both atomic and composite UI components, and real text or image content. Placeholder pages, loading states, error pages, skeleton screens, empty states, and lorem-ipsum pages are removed.

• Application-level complexity. Each retained application must form a genuine multi-route application rather than a set of near-duplicate pages. Reviewers filter out pseudo multi-page apps, require sufficiently rich screenshot-implied interactions across multiple IIS categories, and check that the app renders stably at the fixed capture resolution without overflow, unhydrated content, or dev-server artifacts.

• Corpus-level diversity. The final collection is selected to ensure diversity across application categories, subcategories, and design languages. Reviewers avoid repeated visual themes, duplicated upstream starters, and near-identical templates, so that evaluation reflects model capability rather than memorization of a small set of recurring layouts or scaffolds.

3.2.0.3 Capture pipeline.

Each application is captured by a headless-browser script at \(1440{\times}900\), with timing and content-validation rules producing post-hydration screenshots free of dev-server overlays (full pipeline in Appendix 6). Captures are deduplicated by perceptual hashing and expert-reviewed.

3.2.0.4 Stratification and statistics.

Applications are organised into four application categories by user-facing function: Content, Admin, Transaction, and Specialty. Subcategory listings appear in Table 7 (Appendix).

3.3 Evaluation Protocol↩︎

The protocol decomposes evaluation into a sequence of automated stages for reproducibility, reserving a final human-annotation stage for behavioral judgments that cannot be reliably captured by runtime automated probes. Figure 3 illustrates the full chain. (1) Plan & Generate splits generation into a file-list pass followed by per-file content. (2) Build & Render attempts a production build and a home-route render, producing the per-attempt EXEC pass/fail signal. (3) Self-Debug on EXEC failure feeds the build’s error messages back to the same model for up to three repair attempts. (4) Coverage maps each input screenshot to its generated-app route and emits the page pairs consumed by VFS. (5) Visual Fidelity computes DOM-to-DOM similarity on the matched pairs, defining VFS. (6) Human annotation verifies navigation reachability for NRS and labels each interaction along the interaction taxonomy.

3.4 Standard Metrics: EXEC @, NRS, VFS↩︎

3.4.0.1 EXEC k @k (Executability).

EXEC

@ measures whether the generated source code actually runs, the protocol’s entry gate. An app passes when the production build completes without error and the home route renders meaningful content. EXEC
1
@1 is the one-shot pass rate. EXEC
3
@3, the pass rate after up to three rounds of error-feedback retry, separately measures iterate-fix capability.

3.4.0.2 NRS (Navigation Reachability Score).

NRS measures route-level connectivity: the fraction of input-screenshot routes whose pages are reachable from the home route via visible navigation in the generated application, \[NRS_{a,m} \;=\; \min\!\Bigl(1,\;\frac{n^{\text{reach}}_{a,m}}{n^{\text{tot}}_{a}}\Bigr), \qquad NRS_m \;=\; \frac{100}{N}\sum_{a=1}^{N}NRS_{a,m}, \label{eq:nrs}\tag{1}\] where \(n^{\text{tot}}_{a}\) is the number of input screenshots of application \(a\), and \(n^{\text{reach}}_{a,m}\) counts how many of these routes are reachable in model \(m\)’s artifact. Human verification is used because such navigation (nested menus, modal-triggered routing, conditional links) cannot be reliably enumerated programmatically.

3.4.0.3 VFS (Visual Fidelity Score).

VFS scores what is rendered where via judge-free block-level matching: visible content blocks from each page are paired by optimal bipartite matching, and four sub-metrics (Size, Text, Position, Color) on the matched pairs are averaged unweighted. This design is robust to implementation differences in the underlying DOM. Algorithm and sub-metric formulas in Appendix 11.

3.4.0.4 Conventions.

URL-direct loading: VFS and IIS are evaluated by loading each route directly via URL, isolating page implementation from routing. Zero-imputation: apps failing EXEC
3
@3 contribute zero to NRS/VFS/IIS, keeping models with different EXEC
3
@3 rates directly comparable.

3.5 Interaction Inference Score↩︎

3.5.0.1 Definition.

IIS quantifies how well a model infers and realizes screenshot-implied interactions, jointly considering interaction coverage, implementation result, and state-logic complexity. Rather than comparing against a single reference behavior, IIS adopts a rubric-based evaluation in which each interaction is judged against the functional criteria. This implementation-agnostic formulation accommodates diverse yet semantically equivalent realizations of the same screenshot-implied interaction while enabling consistent evaluation across different applications.

3.5.0.2 Interaction taxonomy and axes.

To make interaction inference measurable, we organize interactions commonly found in web applications into seven categories: toggle, expand/collapse, list operations, data CRUD, form validation, notification, and cross-route state (C01–C07, cataloged in Appendix 7.1). Each category is evaluated along three axes: coverage, indicating whether the interaction category is implied by the screenshots; result, indicating whether the generated application realizes the interaction category (fully realized, partially realized, or failed); and scope (\(S1\) UI-state, \(S2\) data-state, \(S3\) cross-route persistence), reflecting increasing state-management complexity in real front-end engineering. Separating these axes enables failures to be attributed to specific interaction types and aspects, rather than being collapsed into a single aggregate score.

3.5.0.3 Scoring and aggregation.

For each application, we first construct a Reference Interaction Inventory (RII) from the input screenshots by labeling each interaction category with its coverage (whether the interaction is implied by the screenshots) and, when covered, its reference scope \(s^{\text{ref}}_i\). The covered categories form the reference set \(G_a\), ensuring that only interactions inferable from the screenshots are evaluated. For each generated application, every category is annotated along the same three axes: coverage \(y_i \in \{0,1\}\), implementation scope \(s^{\text{gen}}_i \in \{1,2,3\}\), and result \(r_i \in \{1, 0.5, 0\}\) (working, partial, or failed). Each scope level carries a linear weight, \(w_{S1}=1\), \(w_{S2}=2\), \(w_{S3}=3\). Combining the three axes yields the per-application quality score and the model-level IIS: \[\mathrm{QS}_{a,m} = \frac{\sum_{i \in G_a} y_i \cdot \min\!\bigl(w_{s^{\text{gen}}_i},\;w_{s^{\text{ref}}_i}\bigr)\,r_i}{\sum_{i \in G_a} w_{s^{\text{ref}}_i}},\qquad IIS_m = \frac{100}{N} \sum_{a=1}^{N} \mathrm{QS}_{a,m}, \label{eq:iis}\tag{2}\] with \(N\) the number of applications. The denominator is the sum of reference scope weights, which makes IIS recall-oriented.

3.5.0.4 Annotation and validity.

Each generated application is evaluated over all seven interaction categories, yielding 315 IIS items per model and 1,890 annotated items across the six evaluated models. Two front-end engineering experts annotate the reference side to construct the RII. Three annotators independently evaluate the generation side by running the generated applications, exercising each interaction category, and scoring the observed runtime behavior, with disagreements resolved through arbitration. Manual annotation remains necessary because runtime probing alone cannot reliably determine whether different implementations satisfy the same interaction rubric. Inter-annotator agreement is substantial (Krippendorff \(\alpha\) in \(0.72\)–\(0.84\)). Further details are provided in Appendix 7.

4 Experiments↩︎

4.1 Experimental Setup↩︎

We evaluate six frontier VLMs: Claude Sonnet 4.6 [31], Kimi K2.5 [32] (thinking mode enabled), GPT-5.4 [33], Gemini 3.1 Pro Preview [34], Qwen3.5-397B-A17B [35], and GLM-4.6V [36]. Each model is queried through its vendor’s official API. To study how performance scales within a single model family, we additionally evaluate a Qwen2.5-VL ladder (3B, 7B, 32B, 72B) under the same protocol.

4.2 Main Results↩︎

4.2.0.1 Overall ranking across the four metrics.

Table 2 shows the four-metric panorama. Claude Sonnet 4.6 leads EXEC
1
@1 (\(95.6\%\)) and ties Gemini 3.1 Pro Preview at EXEC
3
@3 (\(100\%\)). Gemini leads VFS (\(78.1\)) and Claude leads IIS (\(39.3\)), while GLM-4.6V is lowest on every metric. Among the four metrics, IIS is where the six models diverge most sharply.

3pt

4.2.0.2 EXEC self-debug heterogeneity.

Self-debug benefit varies sharply across models (Table 2, \(\Delta\) column). GPT-5.4 recovers the most (roughly \(+18\) points), Gemini and Qwen3.5 recover moderately, while Kimi shows essentially no gain and GLM-4.6V recovers only marginally (\(+2.2\)); Claude is already ceiling-bounded. EXEC
1
@1 alone therefore substantially understates models that depend on error-feedback recovery, motivating the dual report. Despite its large EXEC

@ gain, GPT-5.4 still ranks \(5/6\) on IIS, suggesting that self-debug improves build success rather than interaction inference.

4.2.0.3 VFS spread is largely build-driven.

Conditioning on EXEC
3
@3-passing apps (VFS\(^{\star}\), Table 5) shrinks the headline VFS range to under a \(20\)-point gap and lifts GLM-4.6V’s visual-fidelity score from \(22.6\) to \(59.8\). The headline spread is thus largely build-driven; the remaining VFS\(^{\star}\) differences trace to the visual-attribute sub-metrics (Size/Text/Position/Color), with the per-component decomposition reported in Appendix 10.

4.3 Interaction Inference Deep Dive↩︎

Through a fine-grained analysis of IIS across interaction categories and scope tiers (Figure 4, Table [tab:per95scope95iis]), we derive the four findings below.

4.3.0.1 Finding 1 — Visual fidelity does not imply interaction-inference capability.

The VFS leader and the IIS leader are different models: Gemini 3.1 Pro Preview ranks first on VFS (\(78.1\)) but fourth on IIS (\(7.5\)), while Claude Sonnet 4.6, second on VFS at \(75.7\), leads IIS at \(39.3\) (\(5.2\times\) ahead of Gemini). Per-cell row-means (Figure 4) put Gemini at exactly \(0.0\) on C04 (CRUD), C06 (notification), and C07 (cross-route state), showing that pixel-faithful reconstruction can still miss core interaction logic.

4.3.0.2 Finding 2 — The closed-vs-open quality hierarchy does not transfer to IIS.

Two of the three open-weight entries (Kimi \(20.7\), Qwen3.5 \(13.2\)) outscore the two closed-frontier entries below Claude (Gemini \(7.5\), GPT-5.4 \(6.7\)). The split is IIS-specific, since Gemini leads VFS and ties Claude on EXEC
3
@3 but trails only on IIS. The closed-vs-open ordering observed on VFS therefore does not carry over to interaction inference.

4.3.0.3 Finding 3 — Cross-route state is a frontier-wide gap.

\(S\)-restricted IIS (Table [tab:per95scope95iis]) decreases as scope becomes harder: \(S1> S2> S3\) holds for nearly all models. The \(S3\) end is a shared frontier limitation: three of six models score exactly zero and only Claude meaningfully clears the floor (at \(21.6\)). Between-model spread also roughly halves as scope hardens, contracting from \(S1\) to \(S3\). \(S3\) remains the benchmark’s most saturation-resistant signal as model families improve.

4.3.0.4 Finding 4 — Admin apps are the consistent IIS bottleneck.

Admin apps yield the lowest six-model mean IIS (\(9.4\)), roughly half of Content and Transaction. This gap is not explained by item count or cross-route incidence, since Transaction has comparable item count and more cross-route state but scores higher. The narrow model spread on Admin further suggests a shared limitation across current models.

4.4 Failure Analysis↩︎

Across the six frontier models, distinct failure mechanisms emerge on the executability, navigation, and interaction dimensions. EXEC
3
@3 losses are dominated by scaffold-respect violations: \(50\%\) of failures violate one of four explicit plan-prompt constraints. Instruction compliance ranges from \(100\%\) (Claude / Gemini / GPT-5.4) down to \(\sim\) \(53\%\) (GLM-4.6V). NRS reductions trace to two recurring patterns: mismatched navigation (a link points to no registered route) and hallucinated routes (pages absent from the input screenshots). IIS zeros concentrate on \(S2\) CRUD and \(S3\) cross-route state. More detailed failure analysis is provided in Appendix 12.

4.5 Within-Family Scaling (Qwen2.5-VL)↩︎

We complement the cross-family comparison with a within-family scaling analysis on the Qwen2.5-VL ladder (3B, 7B, 32B, 72B). All four checkpoints are queried through the vendor’s official API with the same prompt templates. We report the three automatic metrics (EXEC
1
@1, EXEC
3
@3, VFS). NRS and IIS are not collected on this ladder because their human-evaluation cost over four additional ladder points exceeds our annotation budget. Macro means use zero-imputation for EXEC
3
@3 failures, consistent with our main results.

4.5.0.1 Non-gradual scaling: phase transition between 32B and 72B.

Across the \(24\times\) size range, EXEC

@ and VFS are essentially flat below 32B and jump sharply at 72B (Figure 5 (a)): EXEC
3
@3 from \(\le 2.2\%\) to \(62.2\%\), VFS from \(\le 0.2\) to \(35.2\). Usable React-app generation thus emerges between 32B and 72B and is absent below it. Even at 72B, \(37.8\%\) of generations still fail to build, indicating that within-family parameter scaling alone does not saturate UI2App’s executability requirement.

4.5.0.2 Build-failure composition shifts with size, not just rate.

The dominant build-error category shifts monotonically with size (Figure 5 (b)). The smaller checkpoints fail at basic source-code competence (3B on syntax, 7B on inconsistency), while the larger ones fail in more semantically loaded ways (32B on a mix of syntax and hallucinated dependencies, 72B at deeper build stages). Each successive size escapes the previous bottleneck only to surface a more semantically loaded one, a pattern EXEC

@ pass-rate alone does not reveal. Per-size counts and interpretation caveats are in Appendix 12.2.

5 Conclusion↩︎

In this paper, we introduced UI2App, the first benchmark to measure interaction inference: recovering application behavior from image-only multi-page screenshots, with no textual or demonstration specification. Each artifact is scored along four dimensions (executability, navigation reachability, visual fidelity, and IIS), with IIS grounded in the interaction taxonomy that tiers interactions by state-logic complexity. Across six frontier vision-language models, visual fidelity proves a poor proxy for interaction realization: the VFS leader trails the IIS leader by \(5.2\times\). Cross-route state persistence is a frontier-wide bottleneck, with three of six models scoring exactly zero and the strongest reaching only \(21.6\) on \(S3\)-scope interactions. UI2App provides a testbed for closing this \(S3\) headroom. We hope it will drive vision-language models toward both stronger inference and more faithful realization of screenshot-implied interactions.

6 Dataset construction details↩︎

6.0.0.1 Source-pool filter (§3.2).

The four-stage filter referenced in the main text applies the following per-stage criteria to each of the \(2{,}013\) candidate repositories:

1. Permissive license. The project’s upstream LICENSE file (not package-manager metadata) must be MIT, Apache-2.0, BSD, or ISC.

2. Structural validity. The repository exposes a recognizable mainstream front-end framework layout with detectable component and route directories.

3. Buildability. Dependency installation and the production build both complete within a \(180\)-second timeout on a clean environment.

4. Authentication-wall detection. Static heuristics over README/config files combined with HTTP-redirect probing of the served root.

Repositories surviving all four stages enter expert review. The cumulative pass rate is \(164/2{,}013 \approx 8\%\).

6.0.0.2 Screenshot capture pipeline.

Captures use Playwright with headless Chromium at viewport \(1440{\times}900\) and device_scale_factor=1. The four-stage timing waterfall referenced in §3.2 is:

1. domcontentloaded plus a \(3\)-second hydration settle;

2. injection of dev-overlay suppression CSS (Next.js HUD, Vite, webpack);

3. wait_for_function polling for asset readiness (\(6\)-second timeout);

4. animation completion via document.getAnimations() (\(3\)-second timeout), followed by a \(13\)-selector loading-indicator detector with a \(2\)-second recheck.

A per-route login-redirect detector complements the repo-level authentication-wall gate. Surviving screenshots are deduplicated by perceptual hashing on the content area (\(32{\times}32\) average-hash, Hamming distance \(<6\%\)).

7 IIS taxonomy, and inter-annotator agreement↩︎

7.1 Category catalog↩︎

The seven interaction families spanned by IIS and their typical scopes are listed below.

7.2 State management and scope design↩︎

In production front-end engineering, interactive state is managed at increasing levels of architectural scope. The cheapest is component-local state (e.g., React useState/useReducer): state owned by a single component, bound to its mount lifecycle and not shared externally, such as whether a dropdown is open. When a piece of state is read or mutated by several sibling components, it must be lifted to a shared owner (a parent component, a React Context, or a store) so that updates stay consistent across the subtree; typical cases are a list with an active filter, or a selection shared between a toolbar and a table. The most demanding tier is global state that must survive client-side route changes: a single-page app re-mounts the route subtree on navigation, so any state that must persist (a cart, an auth session, a draft) cannot live in the page that gets unmounted and must instead live in a global store whose lifecycle is independent of routing (a Context at the app root, Redux/Zustand/Jotai, or storage-backed). Implementation cost broadly increases along this hierarchy (a local handler, then cross-component state lifting, then a routing-independent global store), and the difficulty of inferring the requirement from the screenshots alone rises with it.

We turn this hierarchy into the three-level scope axis used by IIS, with an operational classification rule. An interaction is S1 (UI-state) when its effect is confined to the immediate rendered subtree (e.g., a toggle); S2 (data-state) when it performs a collection-level transformation (filter/sort/group/CRUD) that other components must observe; and S3 (cross-route persistence) when its effect must outlive a route change (e.g., a cart that stays non-empty after navigating to /checkout).

Because implementation cost broadly rises across S1\({<}\)S2\({<}\)S3, we weight each item by its scope with the linear ordinal weights \(w_{S1}{=}1\), \(w_{S2}{=}2\), \(w_{S3}{=}3\) in Eq. 2 , the simplest choice that respects this ordinal ranking without imposing an arbitrary curvature. The six-model IIS ordering is insensitive to this choice: recomputing model-level IIS under equal weights (\(1{:}1{:}1\)), the default (\(1{:}2{:}3\)), and a ratio-preserving alternative (\(1{:}2{:}4\)) leaves the full ranking unchanged (Table 3; Kendall \(\tau{=}1.0\) between every pair of weightings).

7.3 Outcome rubric and paired-merge interpolation↩︎

For each cell, each annotator assigns the categorical outcome label W (working), P (partial), or F (failed), defined as: W, every behavior the category implies is realized at runtime; P, some implied behaviors are realized while others are not, or one is only partially realized; and F, none of the implied behaviors are realized (the affordance produces no effect). These map to the per-item score \(r_i \in \{1, 0.5, 0\}\) used in Eq. 2 . Under the paired design (§7, two annotators per cell), the cell’s \(r_i\) is set as follows: agreement uses the agreed value; a one-step disagreement (W/P or P/F) is merged by interpolation to \(0.75\) or \(0.25\), so that a partial implementation flagged by one annotator does not collapse to an extreme; a two-step disagreement (W/F) triggers arbitration by the third annotator, after which \(r_i\) takes the arbitrated label.

7.4 Inter-annotator agreement↩︎

Three annotators (A, B, C) labeled in a paired design: \(15\times\), \(15\times\), \(15\times\). Arbitration was triggered on coverage mismatch, result gap \(>1\), or scope gap \(>1\). Agreement values across the four labeled annotation fields are summarized in Table 4. \(95\%\) bootstrap CIs are computed by resampling the \(45\) paired-labeled (app, model) runs (\(15{+}15{+}15\)) with \(B{=}1{,}000\), seed \(42\).

8 Provider-rejected projects↩︎

For GPT-5.4, one project was rejected by the OpenAI content-safety filter before the model received any input screenshots. The main table (Table 2) counts the rejected project as \(0\) in EXEC
1
@1, EXEC
3
@3, and VFS (zero-imputation, denominator \(N{=}45\)). Two alternative conventions yield: (i) \(N{=}44\) zero-imputation (rejected project excluded from the denominator; remaining build/runtime failures still scored \(0\) under zero-imputation): EXEC
3
@3\({=}84.1\%\) (\(37/44\)), VFS\({=}66.4\); (ii) \(N^{\star}{=}37\) conditional on \(EXEC\ifx\\3\\\else @3\fi\)-pass (the VFS\(^{\star}\) convention restricted to the \(44\) evaluable projects). GPT-5.4’s \(EXEC\ifx\\3\\\else @3\fi\) ranking and the broad strong/weak tiering are invariant across all three conventions (it stays fourth, between Kimi and Qwen).

9 Conditional metrics: VFS\(^{\star}\) and IIS\(^{\star}\)↩︎

9.0.0.1 Two denominator conventions.

UI2App reports two complementary scores for VFS and IIS. The headline scores (main table) average over the full curated set \(N{=}45\) with zero-imputation: an application that fails to build after three self-debug rounds (\(EXEC\ifx\\3\\\else @3\fi{=}\mathrm{F}\)) contributes \(0\) to the model’s average, penalizing buildability and the underlying capability jointly. This is the most direct measure of end-to-end usability. The conditional variants VFS\(^{\star}\) and IIS\(^{\star}\) restrict the average to the \(N^{\star}\) projects that pass executability (\(EXEC\ifx\\3\\\else @3\fi{=}\mathrm{P}\), i.e., the build and home-route render both succeed), isolating visual-language understanding and interaction-inference capability from buildability. We report both because the two factor different concerns into the same score:

• Headline (zero-imputation, \(N{=}45\)): is the rendered output of a complete model run usable on average?

• Conditional (VFS\(^{\star}\)/ IIS\(^{\star}\), \(N^{\star}{\le}N\)): given the model produced something at all, how visually faithful or interaction-faithful is it?

Table 5 reports VFS/VFS\(^{\star}\) and IIS/IIS\(^{\star}\) per model alongside their headline\(\to\)conditional gaps. The headline\(\to\)conditional gap is exactly the contribution of build failures to the headline score. Models with no build failures (Sonnet 4.6, Gemini 3.1 Pro Preview at EXEC
3
@3100%) have \(\Delta{=}0\) by construction; high-build-failure models open a large gap. GLM-4.6V’s \(+37.2\)-point gap on VFS (\(22.6 \to 59.8\)) is the most extreme: it has substantial visual-language ability (\(N^{\star}{=}16\) projects average to \(59.8\)), but only \(35.6\%\) of its outputs build at all, so its end-to-end VFS sits last. On IIS, the conditional score IIS\(^{\star}\) similarly lifts GLM-4.6V from headline-last to mid-pack (\(4.5 \to 12.6\)). The top three preserve their order (Claude \(39.3 \to 39.3\), Kimi \(20.7 \to 23.9\), Qwen3.5 \(13.2 \to 17.0\)); the bottom three reshuffle because GLM’s small \(16\)-app build-pass subset yields a higher per-app IIS than its full-set headline, moving it past Gemini (\(7.5 \to 7.5\)) and GPT-5.4 (\(6.7 \to 8.1\)) on the conditional view. The top-three preservation confirms that the IIS gaps among the strongest models in Table 2 reflect interaction-inference capability rather than executability alone.

4pt

9.0.0.2 Choice for the main table.

We headline VFS and IIS (zero-imputation) in Table 2 because end-to-end usability is the property the benchmark targets: a generation run that does not build is unusable to a real user, and its visual or interaction quality on hypothetical-builds-only is not actionable. VFS\(^{\star}\) and IIS\(^{\star}\) are reported as diagnostics that isolate visual-language and interaction-inference capability from build success: they measure output faithfulness over only the projects a model builds, independent of how often it builds.

10 Conditional sub-metric breakdown (VFS\(^{\star}\) decomposition)↩︎

The four zero-imputation VFS sub-metrics (Size, Text, Position, Color) are reported as columns of the main results table (Table 2); \((SIZE{+}TEXT{+}POSITION{+}COLOR)/4\) reconstructs the headline VFS exactly. That zero-imputation view conflates intrinsic visual sub-skill with build success: Gemini 3.1 Pro Preview leads on all four sub-metrics in line with its VFS lead, and GLM-4.6V is lowest on all four, reflecting that its weak VFS stems from buildability (\(35.6\%\)) rather than a sub-skill-specific deficit. Table 6 reports the conditional decomposition that isolates sub-skill from build success: each sub-metric is averaged across the matched pages of the model’s \(EXEC\ifx\\3\\\else @3\fi\)-pass projects only (\(N^{\star}\) apps, \(n_{\text{pages}}\) pages per model), so it decomposes VFS\(^{\star}\) rather than VFS.

The conditional view reveals two patterns invisible in the zero-imputation main table: (i) Sub-skill gaps narrow once buildability is controlled for: Position spread shrinks from \(58.6\) points (zero-impute) to \(9.4\) points (conditional), Color from \(47.9\) to \(11.9\). Visual sub-skill differences across the six models are several-fold smaller than overall VFS differences would suggest; most of the spread comes from build success, not from visual proficiency. (ii) GLM-4.6V’s signature is strong palette plus weak spatial composition: it is last on Size and Position but first on Color (\(80.8\), ahead of GPT-5.4’s \(77.7\)). The Color advantage is partly a selection-bias artifact (\(n_{\text{pages}}{=}76\) from \(16\) build-pass apps), but the cross-validation across the four sub-metrics within those \(16\) apps still suggests GLM-4.6V allocates representational capacity preferentially to color over geometry.

11 VFS algorithm: DOM-alignment matching and cascade fallback↩︎

11.0.0.1 Overview.

VFS is a fully judge-free metric: no VLM, LLM, or pixel-classifier is invoked at any stage. Given a reference application \(\mathcal{R}\) and a model-generated application \(\mathcal{G}\) on the same input set \(\mathcal{I}{=}\{(s_i,\,r_i)\}_{i=1}^{n^{\text{tot}}}\) (screenshot \(s_i\) at reference route \(r_i\)), VFS is produced by the pipeline of Algorithm 6: (A) extract the route set of \(\mathcal{G}\) from src/App.tsx; (B) boot the generated dev server and render each candidate route in headless Chromium under an auth/gate-dismissal protocol; (C) extract reference-side DOM text-blocks from \(\mathcal{R}\), falling back to OCR on the input PNG when \(\mathcal{R}\) cannot be reached at \(r_i\) (auth-locked routes, missing dependencies); (D) match input screenshots to generated routes by global Hungarian assignment on the \(M{\times}N\) matrix of reference-vs-generated DOM-block alignment scores, with a path-token Jaccard tie-breaker and an absolute floor; (E) score the aligned-block set of each matched pair on four sub-metrics (Size, Text, Position, Color); (F) aggregate to page-, application-, and model-level. A legacy cascade \(T_1\)–\(T_5^b\) matcher is retained as a fallback for the rare case where \(\mathcal{S}_{\text{ref}}\) fails to start (§11.1). All hyperparameters are held constant across all models and applications. Specific values are stated inline at each point of use.

11.0.0.2 Auth and gate handling.

React applications in UI2App routinely guard non-public routes behind authentication or splash gates that, if not dismissed, cause every captured route to render the same login/intro view. We apply three source-derived mitigations on the generated dev server (no per-app human configuration), invoked at Algorithm 6 lines 2–4: (i) localStorage auth seeding: scan src/**/*.[jt]s* for storage call sites whose key matches the regex /(auth|session|login|signin|loggedin|token|user)/i and inject a permissive auth payload via Playwright add_init_script; (ii) in-memory auth form-fill: when (i) finds no localStorage hooks but a useState(false) auth flag exists, navigate to /login, fill synthetic credentials, submit, and switch subsequent navigation to pushState so React state survives; (iii) splash/intro gate dismissal: AST-detect the useState(false) + early return pattern in App.tsx, regex-extract the dismiss-button label, and dismiss on every non-/ route (the home is captured with the gate visible to match the reference). The same localStorage seeding is also applied to \(\mathcal{S}_{\text{ref}}\); routes that seeding cannot bypass fall through to OCR.

11.0.0.3 Render-validity probe.

For each rendered route we evaluate a four-signal probe on #root: all (descendant element count), text (innerText length), visuals (svg/img/canvas/video count), and interactive (button/input/textarea/select/a[href]/[role="button"] count). The route is accepted as rendered if all\({\ge}3\) and at least one of: text\({\ge}20\) chars, visuals\({\ge}2\), or interactive\({\ge}2\). The any-of disjunction (rather than a stricter conjunction such as “\(\texttt{text}{>}10\,\wedge\,\texttt{all}{\ge}5\)”) admits legitimate splash/CTA pages with shallow DOM but meaningful content (e.g., a <div><button>Skip</button><p>...</p></div> Intro component). Routes that fail the probe are recorded as None and become MISS candidates for the matcher.

11.0.0.4 Route-alias deduplication.

After each successful screenshot we hash the page’s innerText+child-count signature and consult a per-app seen_dom_hashes table. If the hash recurs we record the new route as an alias of the canonical route and reuse the canonical screenshot rather than writing a duplicate. This collapses redirects (<Route path="/" element=``<Navigate to="/foo">``>) and HashRouter mirror routes that would otherwise inflate \(n^{\text{tot}}\) and double-count gen content.

11.0.0.5 DOM-alignment matching (default path).

We extract DOM text-blocks for every rendered gen route (cached as \(\mathcal{B}^{\text{gen}}_{\hat{r}}\)) and, on the reference dev server, for every input route (cached as \(\mathcal{B}^{\text{ref}}_{s_i}\)). When the reference DOM at \(r_i\) comes back empty (route auth-locked, dependency missing, build/dev-server failure on \(\mathcal{R}\) at this route), we substitute OCR-derived blocks: Tesseract on the input PNG produces line-level boxes which are normalized to the same \(\{\,\texttt{text},\,\texttt{bbox},\,\texttt{fg\_color},\,\texttt{tag}\,\}\) schema as the DOM extractor (Color is degraded for OCR-derived blocks since fg_color defaults to black). Let \(M\) be the number of inputs and \(N\) the number of rendered gen routes. We build the similarity matrix \[\Sigma_{ij}\;=\;\mathrm{compute\_l3\_page}(\mathcal{B}^{\text{ref}}_{s_i},\,\mathcal{B}^{\text{gen}}_{\hat{r}_j})\,/\,100, \qquad \Sigma'_{ij}\;=\;\min\!\bigl(1,\;\Sigma_{ij}+\beta\cdot\mathrm{Jacc}(\tau(r_i),\tau(\hat{r}_j))\bigr),\] where \(\mathrm{compute\_l3\_page}\) is the four-sub-metric scorer (§Sub-metrics below), \(\tau(\cdot)\) tokenizes a path on [-_/]+, and \(\mathrm{Jacc}\) is set-Jaccard. The token bonus \(\beta{=}0.08\) acts as a tie-breaker when DOM-alignment scores are within \({\sim}0.05\) of each other (e.g., gen /leads beats gen /settings/team for the input labeled leads, both candidates at \(\Sigma{\approx}0.55\)), without overriding strong DOM evidence. We solve the assignment globally with the Hungarian algorithm (scipy.optimize.linear_sum_assignment) on cost \(C{=}1{-}\Sigma'\) padded to the square size \(\max(M,N)\) with cost \(1\), then drop pairs whose underlying \(\Sigma_{ij}\) falls below the absolute floor \(\theta_{\text{DOM}}{=}0.20\) (the input becomes MISS and contributes \(0\) to \(n^{\text{matched}}\)). Each gen route is used at most once across the assignment. This hard cap is equivalent to setting \(K_{\max}{=}1\) (the same parameter that the cascade fallback uses, see §11.1) and prevents a single dynamic route (e.g., a /post/:id render) from absorbing three to five unrelated inputs and over-counting matched routes.

11.0.0.6 DOM block extraction.

We navigate Playwright to the route under evaluation, wait until domcontentloaded, dismiss any detected gate (only on non-/ routes; see auth-and-gate handling above), and idle for \(\delta_{\text{ref}}{=}4{,}000\) ms (reference) or \(\delta_{\text{gen}}{=}3{,}000\) ms (generated). Both renderings use viewport \(1440{\times}900\) to match the reference capture. We then run an in-page TreeWalker over text nodes and emit one block per visible parent element with: (i) lower-cased text trimmed to \(200\) characters; (ii) bounding box normalized to viewport so that \((x,y,w,h)\in[0,1]^2{\times}[0,1]^2\); (iii) the foreground color from getComputedStyle().color; (iv) the parent’s tag name. Non-rendering elements (display:none, visibility:hidden, opacity \(0\)), boxes smaller than \(2{\times}2\) pixels, content more than three viewports below the fold (pixel-space \(y_{\text{px}}{>}3v_h\) where \(v_h{=}900\) px is the viewport height), and text strings longer than \(500\) characters are filtered out before the bbox is normalized to \([0,1]^2\). Adjacent same-row same-tag fragments are merged when within \(0.02\) normalized units, suppressing the spurious split of inline-styled text into multiple blocks.

11.0.0.7 Block alignment within a pair.

Let \(B^{\text{ref}}{=}\{b_i^{\text{ref}}\}_{i=1}^{n}\) and \(B^{\text{gen}}{=}\{b_j^{\text{gen}}\}_{j=1}^{m}\). We build the cost matrix \(C\in\mathbb{R}^{n\times m}\) with \(C_{ij}{=}1{-}\rho(t_i^{\text{ref}},t_j^{\text{gen}})\) where \(\rho{=}\,\)difflib.SequenceMatcher.ratio is the Ratcliff–Obershelp similarity (longest common contiguous matching) \(\rho(a,b){=}2L/(|a|{+}|b|)\) with \(L\) the matched-character count. We pad \(C\) to the square size \(\max(n,m)\) with cost \(1\), solve \((\sigma^*,\pi^*){=}\mathop{\mathrm{arg\,min}}\sum C_{\sigma\pi}\) via the Hungarian algorithm (scipy.optimize.linear_sum_assignment), and retain only assignments whose \(\rho{\ge}\theta_{\text{text}}{=}0.3\). The retained assignment set \(A\) is the basis for all four sub-metrics.

11.0.0.8 Sub-metrics.

For a pair \(p{=}(b^{\text{ref}}_p,b^{\text{gen}}_p)\) let \(a(b)\) denote area, \(c(b){=}(x{+}w/2,\,y{+}h/2)\) the center, \(\mathrm{fg}(b)\) the foreground color, and \(\Delta E_{00}\) the CIEDE2000 color difference computed in CIE Lab via skimage.color. The four sub-metrics are \[\begin{align} SIZE\;&=\;\min\!\Bigg(1,\;\frac{\sum_{p\in A}\tfrac{1}{2}\!\left(a(b_p^{\text{ref}}){+}a(b_p^{\text{gen}})\right)}{\tfrac{1}{2}\!\left(\sum_{i}a(b_i^{\text{ref}}){+}\sum_{j}a(b_j^{\text{gen}})\right)}\Bigg),\\[2pt] TEXT\;&=\;\frac{1}{|A|}\sum_{p\in A}\rho\bigl(t_p^{\text{ref}},t_p^{\text{gen}}\bigr),\\[2pt] POSITION\;&=\;\frac{1}{|A|}\sum_{p\in A}\max\!\Bigl(0,\;1{-}\max\!\bigl(|\Delta c_x|_p,\,|\Delta c_y|_p\bigr)\Bigr),\\[2pt] COLOR\;&=\;\frac{1}{|A|}\sum_{p\in A}\max\!\Bigl(0,\;1{-}\Delta E_{00}\!\bigl(\mathrm{fg}(b_p^{\text{ref}}),\mathrm{fg}(b_p^{\text{gen}})\bigr)/100\Bigr). \end{align}\] We use Chebyshev (max-norm) displacement for Position rather than Euclidean because the four-metric average should treat horizontal and vertical drift symmetrically while remaining bounded in \([0,1]\) without an additional normalization constant. CIEDE2000 is divided by \(100\) since saturated complementary colors produce \(\Delta E_{00}{\approx}100\) in the Lab gamut. Each sub-metric defaults to \(0\) if \(|A|{=}0\) (no aligned blocks).

11.0.0.9 Aggregation.

The page-, application-, and model-level scores are \[\ell^{\text{page}}_p \;=\; 25\,(SIZE{+}TEXT{+}POSITION{+}COLOR), \quad L^{\text{matched}}_{3,a} \;=\; \frac{1}{|P_a|}\sum_{p\in P_a} \ell^{\text{page}}_p,\] \[VFS_{a,m}\;=\;L^{\text{matched}}_{3,a,m}\cdot\frac{n^{\text{matched}}_{a,m}}{n^{\text{tot}}_{a}}, \qquad VFS_m\;=\;\frac{1}{N_{\text{app}}}\sum_{a=1}^{N_{\text{app}}}VFS_{a,m}, \label{eq:vfs}\tag{3}\] with \(N_{\text{app}}{=}45\) (the curated app count; the per-app local \(N\) in Algorithm 6 denotes the number of rendered gen routes and is unrelated). The coverage factor \(n^{\text{matched}}/n^{\text{tot}}\) penalizes models that achieve a high per-page score on a small subset of routes. For example, \(1/12\) matched at \(L^{\text{matched}}_{3}{=}31.7\) degrades to \(VFS{}{=}2.6\) rather than retaining the per-page score. Consistent with the scoring policy of §3.3, applications with \(EXEC\ifx\\3\\\else @3\fi{=}\mathrm{F}\) are zero-imputed at the application level. The conditional variant VFS\(^{\star}\) (Appendix 9) drops the zero-imputation, averaging only over \(EXEC\ifx\\3\\\else @3\fi{=}\mathrm{P}\) applications.

11.1 Cascade fallback (\(T_1\)–\(T_5^b\), used only when \(\mathcal{S}_{\text{ref}}\) fails to start)↩︎

When the reference dev server \(\mathcal{S}_{\text{ref}}\) cannot be started for an application (missing node_modules, install timeout, port collision under heavy parallelism), the DOM-alignment matrix in Algorithm 6 is undefined and we fall back to a path-name + visual cascade. Each input is matched to at most one rendered gen route (\(K_{\max}{=}1\), including dynamic routes) and a hit at tier \(T_t\) skips \(T_{t+1},\dots,T_5^b\).

• Exact route match. The reference route \(r_i\) (lower-cased, slashes/query stripped) equals the generated route’s literal definition or its concretized URL.

• Dynamic-route regex match. Generated routes containing path parameters (:slug, :id) are converted to [^/]+ regex. Catches /product/1 \(\leftrightarrow\) /product/:id.

• Path-suffix match. Largest common trailing-segment overlap \(k{\ge}1\). Admits /dashboard \(\leftrightarrow\) /app/dashboard.

• Synonym structural match. Tokenize on [-_/] and expand by a fixed table of \(32\) synonym groups (e.g.{home, landing, index, overview}; {contact, support, etc, misc, links, elsewhere}; {signin, signup, forgot, otp, verify}); a hit requires non-empty intersection with at least one cross-direction unexpanded-vs-expanded overlap.

• Visual-first phash Hungarian. Build the \(M{\times}N\) phash-similarity matrix on \(16{\times}16\) perceptual hashes of input PNG vs gen screenshot, solve Hungarian on \(1-\mathrm{sim}\), and accept pairs with \(\mathrm{sim}{\ge}\theta_{\text{phash}}{=}0.45\).

• Visual fallback (legacy SSIM+hist). For inputs unmatched after \(T_1\)–\(T_5^a\), compute \(0.7\,\mathrm{SSIM}+0.3\,\mathrm{HistCorr}\) on \(256{\times}160\) resized RGB; accept the best candidate only if it satisfies both the relative margin \(\mathrm{best}{\ge}1.2\bar\mu\) and the absolute floor \(\theta_{\text{vis}}^{\text{abs}}{=}0.55\). The absolute floor blocks “least bad” assignments from a uniformly poor candidate set. For instance, it rejects a zen-keyboard screenshot pairing to /player at SSIM+hist\({\approx}0.45\).

11.1.0.1 Implementation notes and known limitations.

The DOM-alignment default is robust to route renaming (gen / that actually serves Blog content correctly pairs with the reference’s /blog input via DOM evidence rather than path string) and rejects path-name false positives (gen / that serves Blog cannot pair with a reference / that serves a music landing). The token-Jaccard bonus \(\beta{=}0.08\) is small enough that DOM evidence dominates whenever it is decisive, and large enough to break ties in the noisy \(0.5\)–\(0.65\) similarity band typical of dark-theme/icon-heavy apps. The OCR fallback for \(\mathcal{B}^{\text{ref}}\) degrades Color (no real foreground color information) but preserves Size, Text, and Position. Because it is applied reference-side—identical for all six models on a given application—it cannot bias cross-model Color comparisons. This is the maximal signal recoverable for routes the reference dev server cannot reach. The strict \(K_{\max}{=}1\) cap occasionally drops the second of two near-identical inputs (e.g., two distinct product-detail screenshots both matching the same /product/:id). This is a deliberate trade-off that removes the much larger error mode of dynamic routes absorbing semantically unrelated inputs. Because input screenshots in UI2App are URL-reachable static routes (§3.2), the pipeline does not replay click sequences before ExtractBlocks. Interaction-state fidelity is captured by IIS (§3.5) rather than VFS. We report VFS as observed under this protocol. An interaction-aware variant is left to future work.

12 Failure taxonomy details↩︎

This appendix expands the failure analysis of §4.4 into a breakdown across the benchmark’s four scoring axes. We first decompose the \(52\) unrecovered EXEC
3
@3 build/runtime failures into a nine-mode taxonomy (Table ¿tbl:tab:failure95modes?) with per-model incidence, prompt-clause attribution, and representative code anchors, then trace how the dominant mode shifts along the Qwen2.5-VL size ladder. We next specify the fixed self-debug feedback protocol under which EXEC
3
@3 is measured, and the recurring patterns that reduce NRS, VFS, and IIS. A closing set of case studies illustrates the latent-affordance inference that IIS targets. Counts are over the \(270\) nominal (app, model) generation pairs spanning \(45\) apps and \(6\) baselines; one additional (GPT-5.4, app) pair rejected by the vendor’s content-safety filter is treated separately (Appendix 8) and is excluded from these \(52\) failures.

12.1 Generation-level failures↩︎

The \(52\) unrecovered EXEC
3
@3 failures decompose into nine fine-grained modes (Table ¿tbl:tab:failure95modes?), which collapse into five high-level causes: hallucinated imports (C1, C6); scaffold-file overwrites (C2, C8); undeclared / wrong-name source dependencies (C3); broken import paths (C4, C5); and parse-level or runtime errors (C7, C9).

4pt

12.1.0.1 C1: hallucinated lucide-react icon export (\(37\%\) of failures).

The model imports a symbol not exported by the scaffold-pinned lucide-react@^0.460.0: typically a brand icon (Tiktok, Spotify, Weibo, Alipay, etc.), a future-renamed name (Funnel, the post-0.460 canonical name introduced when the icon previously exported as Filter was renamed), or a foreign-library identifier (DashboardOutlined from Ant Design). Anchor: GPT-5.4 on 55_shadcn-nextjs-dashboard writes import { Funnel } from "lucide-react", but in 0.460.0 the icon is still exported as Filter.

12.1.0.2 C2: scaffold package.json overwritten (\(35\%\) of failures).

The model emits a package.json in violation of the “Do NOT include scaffold files” clause, pinning unpublished versions or inventing package names (e.g., mantine as a top-level package; registry name is @mantine/core). pnpm install aborts with ERR_PNPM_NO_MATCHING_VERSION / ERR_PNPM_NO_VERSIONS. Anchor: GLM-4.6V on 60_mantine-dashboard.

12.1.0.3 C3: source uses pkg not in extra_dependencies (\(10\%\)).

The plan-step prompt requires a structured extra_dependencies field for non-scaffold packages, but the source imports a package neither pinned in the scaffold nor declared. Declaring non-scaffold dependencies is the model’s responsibility under the plan-step contract, and the Self-Debug stage does not backfill the missing declaration by scanning generated source. Anchor: GLM-4.6V on 54_slash-admin declares react-calendar but imports react-big-calendar.

12.1.0.4 C4: @/* path alias used (\(4\%\)).

The model writes shadcn-style imports (import X from "@/components/X") although the prompt requires relative imports and the scaffold’s tsconfig.json declares no such alias.

12.1.0.5 C6: cross-file export name mismatch (\(6\%\)).

The importer references an identifier whose source declaration uses a different name (default vs.named, casing mismatch, or omitted altogether). Anchor: GLM-4.6V on 38_dexkit-nft-marketplace uses <CollectionCard> at Home.tsx:31 without a matching import. vite build passes (esbuild does not type-check undefined identifiers) but the home route throws ReferenceError on first render, failing EXEC

@’s home-route render gate (§3.4).

12.1.0.6 Long-tail modes C5, C7, C8, C9 (\(10\%\) combined).

C5 (wrong relative path): file-root-relative specifier instead of ../. C7 (JSX/TS parse error): unbalanced closing tag, stray >. C8 (scaffold src/index.css overwritten): @layer base without @tailwind base. C9 (runtime API misuse): GPT-5.4 on 18_myPortfolio destructures a non-existent location field from the <NavLink className={...}> render-prop, which in react-router-dom@6 provides { isActive, isPending, isTransitioning }.

12.1.0.7 Per-model signature and prompt-clause attribution.

Cases are assigned to the earliest-firing root cause: install-time (C2) \(>\) build-time \(>\) runtime (C9); module-resolution (C3, C4, C5) \(>\) symbol-export (C1, C6) within build. Per-model totals (Table ¿tbl:tab:failure95modes?) reveal distinct signatures: Claude and Gemini produce zero failures; GPT-5.4 (\(16\%\) of its \(45\) apps) is dominated by C1 (\(71\%\) of its failures, all factuality); GLM-4.6V (\(64\%\) of its apps) is dominated by scaffold-respect violations (C2+C3+C4+C8 \(\approx 72\%\)). The plan-step prompt (Appendix 13) states four mechanically-checkable constraints: (i) source files under src/; (ii) relative imports; (iii) every non-scaffold import declared in extra_dependencies; (iv) no regeneration of scaffold files. Half of all failures are instruction-following violations (IFV column of Table ¿tbl:tab:failure95modes?): clause (iv) accounts for \(73\%\) of IFVs via C2 + C8, clause (iii) for \({\sim}20\%\) (C3), clause (ii) for \(8\%\) (C4), and clause (i) has zero violations. The other half (C1, C5, C6, C7, C9) are coding-quality or factuality errors not pre-empted by the prompt. The Instruction-Compliance Score (ICS) row shows the resulting model-level split. Self-debug retries a failing app up to three rounds with the same model and a fixed feedback window. An app failing after round \(3\) is scored EXEC
3
@3F under this fixed, model-identical feedback protocol (Appendix 12.3).

12.2 Per-size failure-mode shifts on the Qwen2.5-VL ladder↩︎

Figure 5 (b) composes the build-failure category at each size, supporting the dominant-shift narrative in §4.5: the dominant category shifts from syntax (3B) to inconsistency (7B) to a mixed syntax \(+\) hallucination regime (32B). The post-transition \(72\)B residual is dominated by syntax at deeper build stages plus the figure’s other bucket, here runtime errors (e.g., importing TypeScript-only types as values). These four buckets correspond to the C1–C9 taxonomy of §12.1 as: syntax\(\supseteq\)C7, inconsistency\(\supseteq\), hallucination\(\supseteq\), other\(\supseteq\)C9. Each successive size escapes the previous bottleneck only to surface a more semantically loaded one. We caution that the breakdown at \(\leq 32\)B is over a near-fully-failing population (\(\geq 44/45\) failures per size), so the precise within-size category proportions are noisy; the identity of the dominant (mode) category, taken over \(44\)–\(45\) failures, is well determined and is what the cross-size dominant-shift claim relies on.

12.3 Self-debug feedback protocol↩︎

EXEC
3
@3 is measured under a Self-Debug stage configured identically for every model: each failing app is retried for up to three rounds with the same model, and each round forwards the toolchain’s own diagnostics back to the model for repair. EXEC
3
@3 therefore reflects iterate-and-fix ability under this fixed feedback protocol rather than the model in isolation; how reliably a model localizes and repairs the root cause from standard build output is part of what it measures, with weaker self-debuggers returning NO_FILE.

12.4 Navigation and visual-fidelity failures↩︎

Among EXEC
3
@3-passing apps, NRS is reduced by two recurring patterns: (i) mismatched navigation, where a <Link to="/foo"> resolves to no <Route> in the application’s router, and (ii) hallucinated routes, where the application registers and renders a route that corresponds to no input screenshot. Mismatched navigation dominates: models default to plausible web-app conventions such as sign-in, sign-up, forgot-password, or dashboard sub-routes when the input screenshot set underspecifies the route closure.

Among \(EXEC\ifx\\3\\\else @3\fi\)-pass apps, the sub-skill view is best read off the conditional decomposition Table 6 (the zero-imputation columns of Table 2 are dominated by build-success differences, not sub-skill). Conditional spreads are tight: Color \(68.9\)–\(80.8\) (\(11.9\)), Position \(73.9\)–\(83.3\) (\(9.4\)), Size \(70.6\)–\(75.6\) (\(5.0\)), Text \(87.0\)–\(91.7\) (\(4.7\)). Position drops are dominated by reference-subset rendering rather than horizontal grid collapse: the generated DOM contains a strict subset of the reference’s blocks, so unpaired reference blocks dominate the centroid mismatch (see the GLM-4.6V case study in Fig. [fig:case95grid]). Color drops by mis-derived theme palettes when screenshots use a non-default brand color. GLM-4.6V is the salient outlier: last on Size and Position but first on Color among build-pass apps. Palette extraction on the matched subset is strong, but the generated DOM covers only part of the reference.

12.5 Interaction-inference failures↩︎

Figure [fig:int95failure95cart] anchors a representative IIS failure on a shared input. Two models receive the same five e-commerce screenshots of 48_shopco-ecommerce; an annotator then performs the same Add-to-Cart action on the product page and navigates to /cart. Claude Sonnet 4.6 (left) threads the action through a CartContext and the cart route renders the added product (result=W). Gemini 3.1 Pro Preview (right) emits a visually-faithful product page with an Add to Cart button whose handler does not propagate to any cross-route store, so the cart route still renders its empty-state placeholder (result=F). The cart route’s static screenshot at protocol time is therefore indistinguishable from the reference’s empty-cart capture.

12.6 Case studies: latent-affordance inference↩︎

The eight case studies below illustrate four claims from the main text: bimodal interaction emergence, domain-level behavioral inference, build-time failure modes, and VFS sub-metric divergence; they are chosen to span these four claim types, and the aggregate evidence for each lives in the corresponding section and table (§4.3, Tables ¿tbl:tab:failure95modes? and 6).

12.6.0.1 Audio synthesis (31_genshin-music).

The 31_genshin-music task supplies \(9\) screenshots of a stylized in-browser music synthesizer (Fig. [fig:case95genshin95renders]): a hexagonal note pad, a key-binding overlay, a composer timeline, and a full-screen “Zen mode” player. None of the screenshots state explicitly that the application produces sound. The audio capability must be inferred from layout cues alone: musical symbols on each pad, MIDI-style key labels (Q–U / A–J / Z–M), an instrument-picker dropdown, and a decay-envelope visualization behind each note.

Of the six baselines, four reproduce the interface as a runnable static visual replica: pads animate on hover, but onMouseDown handlers are absent or no-ops and the emitted package.json declares no audio dependencies; GLM-4.6V fails to build. Only Kimi K2.5 instantiates a complete Web-Audio signal chain (AudioContext \(\to\) triangle-wave OscillatorNode \(\to\) GainNode with a \(500\) ms exponential-decay envelope) and wires keyboard events together with pointer events on every pad through a MIDI-to-frequency converter \(f = 440 \cdot 2^{(n-69)/12}\). The artifact is a functionally playable synthesizer rather than a screenshot.

The contrast illustrates the regime IIS (§3.5) is designed to expose. The rendered interfaces are visually comparable and VFS cannot separate them, yet only one artifact is functionally playable. Crucially, no input screenshot can be flagged as “wrong” for the four UI-only models, since their renderings reproduce the visual targets’ structure: grid, labels, and layout. The gap is one of behavior inferred from cues that the visual modality under-specifies, precisely the regime UI2App is built to expose for multi-modal code-generation agents beyond surface mimicry. Figure [fig:case95genshin] contrasts the source for one pad cell across the two regimes (the UI-only listing is from Gemini 3.1 Pro Preview, representative of the four visually-faithful-but-silent baselines).

12.6.0.2 State-machine inference (01_fashion-ecommerce, 48_shopco-ecommerce).

Two e-commerce tasks, 01_fashion-ecommerce (\(6\) screenshots: home, shop, product detail, cart, checkout, search) and 48_shopco-ecommerce (\(5\) screenshots: home, shop, cart, and two product details), require an analogous inference along an orthogonal direction. Rather than an audio runtime, the model must infer the underlying state machine that backs the interface: a global cart that survives route changes (S3), variant selection on the product page that propagates into add-to-cart payloads (S2), search and sort on the shop (S2), and controlled inputs on a checkout form. As with audio, the screenshots never declare these affordances explicitly: a “$0.00” subtotal hints at a reactive total, a size grid at selectable variants, a search bar at client-side filtering.

Across interaction items spanning the two projects, Claude Sonnet 4.6 produces lots of interaction components that work end-to-end: a CartContext de-duplicating items by the composite key \((\textit{id},\,\textit{size},\,\textit{color})\), auto-clearing entries when quantity drops to zero, a controlled CheckoutPage form, and a search page with submit-driven filtering on product name and category. The next-best on these two tasks, GPT-5.4, supplies a partially wired ShopContext on 48_shopco-ecommerce, but its addToCart mutator is never read by the cart page, which renders a static empty-state placeholder, dead code in the literal sense. The remaining four baselines (Gemini 3.1 Pro Preview, GLM-4.6V, Qwen3.5-397B-A17B, Kimi K2.5) systematically emit visually faithful but inert components: add-to-cart buttons without onClick, cart pages reduced to “Intentionally left blank” placeholders, and no cross-component state container at all. Figure [fig:case95ecommerce] contrasts the /cart render across the three regimes.

12.6.0.3 Persistent task state (39_maciekt07-todoapp).

The todo task supplies \(5\) screenshots of a single-page task manager with categories, multi-select, completion toggles, and JSON import/export. Of the six models that generated this app, only Kimi K2.5 persists task state, lifting it into a single AppContext that mirrors itself into localStorage on every mutation (STORAGE_KEY = "taskapp_data") and re-hydrates on mount. None of the other five survives a reload: Claude implements working in-memory CRUD in an app-level Context, while GPT-5.4, Gemini, and Qwen render the task list as static mock data with no mutable state; GLM-4.6V’s generated source holds tasks in component-local useState, but the app fails to build (\(EXEC\ifx\\3\\\else @3\fi{=}\mathrm{F}\)). Figure [fig:case95todoapp] verifies the re-hydration half directly: three tasks seeded into localStorage survive a hard reload and re-appear on mount. This cross-session (localStorage-backed) persistence is strictly stronger than the S3 cross-route criterion (§7), which several models satisfy only in memory (e.g., Sonnet’s cart in Fig. [fig:case95ecommerce]).

12.6.0.4 Drag-and-drop file upload (39_maciekt07-todoapp).

The same task’s /transfer screen shows a dashed-border “Drop JSON file here to import tasks” import region. The cue is purely visual: a dashed rectangle with an upload icon and a brief instruction label, with no API specification. Kimi K2.5 and Claude Sonnet 4.6 both extract a FileDropZone component bound to native HTML5 onDragOver/onDrop events that pre-fills the JSON-import field on drop. Gemini 3.1 Pro Preview, GPT-5.4, and Qwen3.5-397B-A17B render the dashed rectangle as decoration plus a click-only fallback button with no drop handlers; GLM-4.6V’s build of this app fails (\(EXEC\ifx\\3\\\else @3\fi{=}\mathrm{F}\)).

12.6.0.5 Rich-text editing (67_shadboard).

67_shadboard ships an editor screen whose only behavioral cue is a formatting toolbar (bold, italic, list, link) above an empty document area. Claude Sonnet 4.6 and Qwen3.5-397B-A17B mark the document area with contentEditable, allowing genuine cursor-driven typing on the rendered page. Kimi K2.5 does the same and additionally binds the toolbar buttons to document.execCommand. GPT-5.4 is the sharper negative: it renders the full editor UI — a formatting toolbar above a document pane — that looks like a working editor, yet the pane is a static <div> (the toolbar icons carry no handler and there is no <textarea> or contentEditable element), so typing produces no change. The failure is indistinguishable from a real editor at idle and surfaces only under the live-typing test (Fig. [fig:case95editor]). The remaining two models land between these poles: GLM-4.6V wires a plain <textarea> that accepts text but leaves its formatting toolbar inert, and Gemini 3.1 Pro Preview emits no /editor route at all.

12.6.0.6 Reference-subset rendering (62_tokyo-mui-dashboard).

The Tokyo crypto-dashboard organizes content into two vertical sections: a top section (Welcome header, Account Balance card, donut chart with per-asset breakdown) and a continuation (Wallets row of three asset cards plus the Account Security side card). Gemini 3.1 Pro Preview reproduces both sections: the Wallets header is visible at the bottom of the captured viewport, with the first row of asset-card frames already laid out. GLM-4.6V’s render is a strict subset of the reference: the top section is reproduced faithfully (the donut, the Account Balance card with the same numeric value and Send/Receive controls, the per-asset legend), while the Wallets row and Account Security card have no counterpart in the generated DOM. We verify the subset relation rather than infer it from a viewport screenshot: a full-page Playwright capture across seven candidate routes (/dashboard, /crypto, /cryptocurrency, /dashboards/crypto, /dashboard-crypto, /overview, /) returns document.documentElement.scrollHeight=900 px (exactly the viewport height, so no overflow exists where missing blocks could hide), and document.body.innerText on every probed route contains none of the Wallets/Security section keywords. The Wallets/Security blocks are absent from the rendered DOM, not pushed past the viewport. This subset relation is exactly what the DOM-block matching procedure of VFS penalizes through the coverage factor \(n^{\text{matched}}/n^{\text{tot}}\) in Eq. 3 : each reference block with no counterpart in the generated DOM is a missed match, and the unmatched-block count dominates Position/Size for this app. The case is the per-app face of GLM’s signature VFS\(^{\star}\) profile (Table 6): the matched subset (top row) keeps palettes close to the input, explaining GLM’s highest aggregate Color, while the unmatched complement (the below-fold sections) keeps GLM’s Position/Size the lowest on the build-pass subset.

12.6.0.7 Build-time failures (55_shadcn-nextjs-dashboard, 60_mantine-dashboard).

The two dominant unrecovered EXEC
3
@3 modes (Table ¿tbl:tab:failure95modes?) have orthogonal etiologies. C1 (hallucinated icon) is a factuality error: GPT-5.4 imports Funnel from lucide-react, but in the scaffold-pinned 0.460.0 the icon is still exported as Filter (Funnel is the post-0.460 canonical name); vite build aborts with “"Funnel" is not exported by …/lucide-react.js”. C2 (package.json overwrite) is an instruction-compliance error: GLM-4.6V regenerates the scaffold’s package.json despite the explicit “Do NOT include scaffold files” clause (Appendix 13, clause iv), pinning the unpublished version "mantine":"^1.0.0"; pnpm install aborts with ERR_PNPM_NO_VERSIONS.

12.6.0.8 Functional quiz engine (61_react-duolingo).

The gamified learning app’s \(10\)-screenshot input set (Fig. [fig:case95duolingo95input]) spans a lesson map, a lesson exercise, a leaderboard, a shop, a profile, and settings. The lesson exercise — a prompt, three answer tiles, and a Check button — has behavior the screenshots never state: selecting a tile, judging it against the correct answer, and awarding XP on success must all be inferred. Across the six baselines the rendered exercise is near-identical, yet only Claude Sonnet 4.6 wires a working quiz engine — handleCheck compares the selection to question.correctId, sets correct/incorrect feedback, and on a correct answer calls incrementLessons (advancing lesson progress and adding \(10\) XP). Kimi K2.5 only partially wires it (its Check handler navigates without validating or scoring), and the remaining four (Gemini 3.1 Pro Preview, GPT-5.4, Qwen3.5-397B-A17B, GLM-4.6V) render a static mockup whose Check button carries no handler. The \(1\)-functional / \(1\)-partial / \(4\)-static split is invisible to VFS — every render shows the same tiles and button — and is the answer-validation analogue of the audio case (Fig. [fig:case95genshin]).

12.6.0.9 Synthesis across cases.

The eight cases together draw three lines that no single metric exposes. First, latent-affordance inference is neither monotonic in model scale nor in visual fidelity: Kimi K2.5 is the only baseline that recognizes the audio runtime in 31_genshin-music (Fig. [fig:case95genshin]) and the only baseline that lifts task state into localStorage on 39_maciekt07-todoapp (Fig. [fig:case95todoapp]), yet it ships inert e-commerce UIs (Fig. [fig:case95ecommerce]). Claude Sonnet 4.6 dominates state-machine inference on both shops and reaches for contentEditable on the editor task (Fig. [fig:case95editor]) and alone wires the answer-validation \(+\) XP engine behind 61_react-duolingo’s lesson exercise (Fig. [fig:case95duolingo]), yet ships a silent music application. Second, the failure surface is itself bimodal (Fig. [fig:case95build], quantified in Table ¿tbl:tab:failure95modes?): GPT-5.4’s unrecovered EXEC
3
@3 losses are dominated by factuality errors (hallucinated icons), whereas GLM-4.6V’s are dominated by instruction-compliance errors (scaffold overwrites), two orthogonal error modes that EXEC
3
@3 alone conflates. Third, the visual-fidelity dimension is internally split (Fig. [fig:case95grid]; aggregate decomposition in Table 6): GLM-4.6V’s Color sub-metric is the highest of any model on the build-pass subset while its Position sub-metric is the lowest, an inversion that motivates layout-targeted training rather than further color supervision. Across all eight cases, the disagreement among baselines on which behavioral type or structural primitive is recognized is precisely the regime that pixel-fidelity scores cannot expose and that IIS, NRS, and the failure taxonomy are jointly constructed to surface.

13 Prompt templates↩︎

Below are the prompt templates used in the pipeline. Placeholders rendered as {name} are populated at run time; line breaks and angle-bracket tags are exactly what the model receives.

Stage 1 (Plan).↩︎

The Plan prompt is issued in the second user turn after the assistant has emitted the scaffold listing in Turn 1. The same turn carries all \(M\) screenshots as image content.

The four constraints we treat as explicit instruction clauses for the failure-mode attribution in §4.4 are: (i) all source files under src/; (ii) relative imports; (iii) every non-scaffold package declared in extra_dependencies; (iv) no regeneration of scaffold files.

Stage 2 (Generate).↩︎

The Generate prompt is issued in the next user turn after the model has returned the plan. Screenshots are not re-sent.

If the model’s response is truncated (finish_reason \(\ne\) stop), we send up to three Continue turns: “Continue writing from exactly where you left off. Do not repeat any content. Continue with the next file or finish the current file.”

Stage 3 (Self-Debug).↩︎

On EXEC failure the same model is re-invoked for up to three rounds. Each round runs three sub-prompts as needed.

13.0.0.1 (a) File-localization fallback.

When the regex extractor cannot identify which source files to repair from the build error, the model is asked to localize:

13.0.0.2 (b) Build-error repair.

For each identified file the model receives the error, the project file listing, and the file’s current contents:

13.0.0.3 (c) Runtime-error repair.

When the build passes but the home route fails to render, the model receives all candidate files (App.tsx, main.tsx, plus any flagged by error parsing) together for a holistic fix:

References↩︎