Abstract

Vision-language models (VLMs) have made strong progress on high-level image-text alignment, yet their ability to perceive subtle visual differences remains limited. We study this problem in rendered web interfaces, where localized visual changes are both a diagnostic test of fine-grained perception and a practical requirement for GUI agents and design tools. We introduce DiffSpot, a code-driven benchmark for open-ended spot-the-difference on web interfaces. DiffSpot constructs controlled image pairs by mutating a single CSS property of a target element in self-contained HTML, re-rendering the page, and recording the changed property, element, and mutation magnitude. A grounding gate retains only pairs whose rendered pixel difference is confined to the target element. The benchmark contains 4,400 pairs, including 3,900 has-diff pairs balanced across 13 CSS-property operators and three difficulty tiers, plus 500 no-diff pairs for hallucination control. Evaluating 13 frontier VLMs zero-shot, we find that even the best model identifies only \(40.7\%\) of true changes, with Hard-tier Recall below \(23\%\) for every model. DiffSpot further shows that difficulty is strongly property-dependent: across CSS operators, neither pixel magnitude nor CLIP distance reliably predicts Recall.

	Code	`https://github.com/Tencent/DiffSpot`
	Data	`https://huggingface.co/datasets/tencent/DiffSpot`

1 Introduction↩︎

Figure 1: DiffSpot construction pipeline. DiffSpot turns real web pages into controlled before/after screenshot pairs by moving visual-difference construction from image space to code space. A. Corpus curation collects and filters rendered web-interface candidates from URL-seeded pages. B. Programmatic mutation applies single CSS-property changes across operator-specific difficulty tiers. C. A grounding gate validates the rendered result, retaining only pairs whose pixel difference is confined to the target element. — Figure 1: **DiffSpot construction pipeline.** DiffSpot turns real web pages into controlled before/after screenshot pairs by moving visual-difference construction from image space to code space. A. Corpus curation collects and filters rendered web-interface candidates from URL-seeded pages. B. Programmatic mutation applies single CSS-property changes across operator-specific difficulty tiers. C. A grounding gate validates the rendered result, retaining only pairs whose pixel difference is confined to the target element.

Vision-language models (VLMs) have made strong progress on high-level image-text alignment and language-guided visual reasoning, yet remain brittle on fine-grained visual perception [1]–[3]. A direct stress test of fine-grained perception is spot-the-difference: given two nearly identical screenshots, can a model tell exactly what changed? In such near-identical pairs, semantic shortcuts provide little help; the model must localize and name a small visual change against an otherwise unchanged background. This capability is not only diagnostic, but also practically necessary for systems that operate on rendered web interfaces, including GUI agents and design tools [4]. Such systems must verify not merely that the screen changed after an action or edit, but which visual property changed, where it changed, and whether the change was confined to the intended element.

Despite this diagnostic and practical need, web-interface spot-the-difference remains largely absent from existing pair-difference benchmarks for VLMs [5], [6]. The bottleneck is twofold. First, it is difficult to collect the right image pairs at scale: real web pages rarely provide near-identical before/after screenshots that differ by only one small UI property, let alone balanced coverage over UI elements, visual properties, and difficulty levels. Second, even when similar pairs are available, post-hoc human labeling is a selective filter rather than a neutral sampler of visual differences. Human change detection is known to be attention-, semantics-, and change-type dependent, rather than determined by pixel magnitude alone [7]–[11]. As a result, annotation tends to over-represent changes that are visually salient, semantically meaningful, or easy to name, while subtle visual-property changes are more likely to be missed, labeled at inconsistent granularity, or unevenly covered across properties and difficulty levels. Thus, the target regime for web-interface diffing—subtle, localized, property-level differences—remains under-covered.

Our key insight is to move difference construction from image space to code space. Unlike natural images, web interfaces are programmatic visual artifacts: their pixels are generated by rendering structured HTML/CSS. This lets us define a visual difference before an image pair is created, rather than discover one after the fact. Given a self-contained HTML page, we mutate a single CSS property of a target element and re-render the page in a headless browser. The resulting before/after screenshots are paired with a machine-readable mutation record specifying the changed property, target element, and mutation magnitude. By varying the mutation magnitude within the same CSS operator, we can parameterize difficulty while holding the visual property fixed. This code-driven construction directly addresses both bottlenecks: it systematically generates near-identical pairs with balanced coverage across UI elements, visual properties, and difficulty levels, while avoiding post-hoc annotation bias by specifying each difference before rendering.

However, specifying differences in code does not guarantee that they appear as clean, localized differences after rendering. A mutation can be ineffective, when the CSS change is shadowed and produces no visible effect, or non-local, when it triggers layout reflow and changes regions beyond the target element. In either case, the mutation record no longer corresponds to a single localized visual effect in the rendered screenshots. To enforce this correspondence, we introduce a grounding-gate mechanism that validates each candidate mutation in rendered pixel space. Using the target element specified in the mutation record, we obtain its browser-rendered bounding box and accept a pair only when the rendered pixel difference is confined to this target region. The retained pairs therefore align each machine-readable mutation record with exactly one effective, localized visual difference at the intended element.

We instantiate this idea in DiffSpot, a construction pipeline that turns real web pages into controlled before/after screenshot pairs. Starting from URLs collected from 2M source domains, we crawl and render pages, convert them into self-contained HTML sources, and apply CSS-property mutations followed by grounding-gate filtering. The resulting benchmark contains 4,400 image pairs: 3,900 has-diff pairs balanced across 13 CSS-property operators and three difficulty tiers (100 pairs per operator–tier cell), plus 500 no-diff pairs for hallucination control.

A zero-shot evaluation of 13 frontier VLMs [4], [12]–[17] shows that fine-grained visual difference detection on web interfaces remains far from solved: even the best model identifies only \(\mathbf{40.7\%}\) of true visual changes, and Hard-tier Recall stays below \(23\%\) for every model. Beyond this low ceiling, DiffSpot reveals a property-level failure pattern. Neither bbox-level pixel change nor CLIP image distance reliably predicts Recall, suggesting that models struggle to perceive and name CSS-level visual properties rather than simply detect larger image differences. Moreover, performance varies much more across CSS operators than across source domains: what changed matters more than where it appeared. The 500 no-diff pairs further expose a sensitivity–restraint trade-off, disentangling sensitivity to real changes from restraint on unchanged pairs.

Our contributions are:

A web-interface spot-the-difference benchmark. We introduce DiffSpot, to our knowledge the first benchmark for open-ended spot-the-difference on rendered web interfaces.
A code-driven visual-difference generation pipeline. We develop a pipeline that creates controlled visual differences by programmatically mutating CSS properties in self-contained HTML and validating the rendered result with a bounding-box grounding gate.
Property-level diagnostic findings. We evaluate 13 frontier VLMs zero-shot and reveal property-specific failures: even the best model identifies only \(40.7\%\) of true visual changes, while pixel- and CLIP-based magnitudes poorly predict Recall.

2 Benchmark Construction↩︎

DiffSpot is built by a five-stage pipeline (Figure 1) that turns a large pool of rendered web pages into a quality-gated benchmark balanced across operator–difficulty cells: source corpus curation (§2.1), programmatic mutation (§2.2), grounding-gate validation (§2.3), polish and filtering (§2.4), and stratified sampling with a no-diff control (§2.5). All ground truth is derived programmatically from the structured mutation record.

2.1 Source Corpus Curation↩︎

We seed the corpus from the Chrome User Experience Report Top-1M and Majestic Top-1M (2M domains; 1.35M after host dedup), expand each domain via its sitemap to 17.75M page URLs, and keep one URL per HTML-structure fingerprint (9.04M structure-unique URLs). Each URL is rendered in headless Chromium driven by Playwright [18] at a fixed \(1280\times800\) viewport and paired with a self-contained HTML produced by LLM regeneration, keeping the released dataset free of third-party licensed content. We retain only pairs whose CLIP [19] similarity between the original and regenerated renders is \(\geq 0.70\), yielding the sampled image–code pair pool shown in Figure 1. A three-VLM-judge realness audit on 501 pairs scores renderings within 0.3 points of originals on a 5-point scale (§3.6). After rule-based content filtering for PII, abnormal HTML body length, and dynamic tags, an LLM domain/style labeler (gpt-oss-120b [20]) and a capped sampling policy yield the multi-label enriched pool that feeds the mutation stage (Figure 1; §11).

2.2 Programmatic Mutation↩︎

DiffSpot is by design restricted to atomic, localized visual differences: each pair isolates a single CSS-property-level mutation on a single target element. This scope is essential for a per-property capability probe. Compound or reflow-heavy changes can introduce coarse visual cues and break the correspondence between the mutation record and the rendered screenshots; the resulting pair no longer isolates which CSS-level visual property a model actually perceives and names.

We define 13 CSS-property-level operators grouped into four families: typography (font_weight, font_size, letter_spacing, line_height, text), color (color, opacity, gradient), layout (position, spacing, justify), and shape (border, rounded). Two mutation mechanisms are selected per operator—a Tailwind-CSS [21] class swap and an inline-style override; both operate on the static HTML and are fully reproducible (§10).

2.2.0.1 Difficulty tiers.

Each operator is stratified into Easy, Medium, and Hard tiers with non-overlapping parameter ranges, so the difficulty tier of a candidate is parameterized solely by the magnitude of the property change. For step-based operators (e.g.rounded, color), tiers correspond to Tailwind-scale step distance (Easy: 3–5, Medium: 2, Hard: 1); for continuous-valued operators (e.g.letter_spacing), tiers are em-offset magnitude (Easy: \(\pm0.20\) em, Medium: \(\pm0.12\) em, Hard: \(\pm0.06\) em). Full parameter ranges are in Table 3 (§12.3). Difficulty is strictly ordered within each operator. Each pair is processed in grouped mode (one candidate per tier) to produce the raw mutation candidate pool, which the grounding gate then validates (Figure 1, Panel C).

Figure 2: DiffSpot dataset statistics. (a) Balanced design. 13 operators \times 3 difficulty tiers = 39 cells with 100 has-diff pairs each (3,900 total), plus 500 no-diff pairs. Color encodes operator family; shade encodes difficulty. (b) Source-domain coverage. All 15 domain categories, sorted by frequency. — Figure 2: **DiffSpot dataset statistics.** **(a) Balanced design.** 13 operators \(\times\) 3 difficulty tiers \(=\) 39 cells with 100 has-diff pairs each (3,900 total), plus 500 no-diff pairs. Color encodes operator family; shade encodes difficulty. **(b) Source-domain coverage.** All 15 domain categories, sorted by frequency.

2.3 Grounding Gate↩︎

Naively re-rendering a mutated HTML can break the intended code-to-pixel correspondence in two ways: no-effect mutations, where the CSS override is silently shadowed and produces no visible effect, and reflow contamination, where the mutation cascades beyond the target element and changes other regions of the page. A full-image pixel-diff filter can verify that the page changed, but not that the change is localized to the intended element. We therefore anchor validation to the target element’s bounding box, queried declaratively from the rendered DOM; the bbox is never inferred from pixel content. The gate requires three predicates:

Effectiveness. Inside-bbox pixel change is non-zero.
Locality. Outside-bbox region is unchanged at pixel level.
Selector resolution. Selectors that fail to resolve in the rendered DOM are rejected.

These predicates jointly retain only pairs in which the rendered pixel change is concentrated inside the target bounding box and absent outside it.

2.4 Polish and Filtering↩︎

We polish each gated record into a natural-language answer using gpt-oss-120b [20] at temperature \(0.7\) with paraphrase exemplars. Each pair ships with both the structured mutation record, which defines the scoring ground truth, and the natural-language description, which is used only for display. A small set of content filters then removes residual quality-failure cases (§11).

2.5 Stratified Sampling and No-Diff Control↩︎

We partition the filtered candidates into 39 cells (13 operators \(\times\) 3 difficulty tiers) and draw exactly 100 candidates per cell (3,900 total). The \(n{=}100\) choice gives a per-cell binomial standard error of \(\approx 5\) pp at \(p{=}0.5\), tight enough for per-cell reporting in §[sec:property95failures]. To measure hallucination, we add 500 no-diff pairs constructed by rendering the same HTML twice (no mutation applied); the ground-truth answer is “No visible differences.”

2.6 Final Composition↩︎

The final DiffSpot benchmark contains 4,400 image pairs: 3,900 has-diff pairs (100 per operator-tier cell, 39 cells) and 500 no-diff pairs. Figure 2 summarises the balanced design and source-domain coverage. Each pair ships with the before/after PNG screenshots, the structured mutation record, and the polished natural-language description; the benchmark is fully regenerable from the released self-contained HTML under a deterministic rendering pipeline.

3 Experiments↩︎

3.1 Setup↩︎

3.1.0.1 Models.

We evaluate 13 recent and frontier vision-language models on DiffSpot zero-shot across proprietary API and open-weight access categories. Proprietary API: four models accessible only through vendor APIs [4], [12], [13], [22]–[24]. Open-weight: nine models with publicly released weights: Kimi K2.5 [15], Qwen3.5-VL-397B [25], GLM-4.6V/-Flash [16], a 2\(\times\)2 Qwen3-VL grid ({30B, 235B} \(\times\) ) [14], and InternVL3.5-30B-A3B [17]. Full names and parameter counts are in Table [tab:main95table]; the 2\(\times\)2 Qwen3-VL grid lets us isolate the effect of reasoning mode at two scales.

3.1.0.2 Inference.

All models use greedy decoding (temperature \(0\)) with a 16,384-token output budget; image pairs are fed at the original \(1280\times800\) viewport resolution (§2). The prompt is a single zero-shot instruction that presents the before/after screenshots and asks the model to list observed differences, without worked examples or hints. The prompt is identical across models.

3.1.0.3 Metrics.

We reduce each open-ended response to a per-case binary verdict and report Accuracy \(= (TP+TN)/4{,}400\), where \(TP\) counts has-diff pairs whose ground-truth mutation is identified and \(TN\) counts no-diff pairs correctly reported as having no change. Sliced views break this score down into Easy / Med / Hard Recall (correct identifications among the 1,300 has-diff cases in each tier) and No-Diff Acc. (\(= 1-\text{hallucination rate}\) on the 500 no-diff pairs). Matching is performed by gpt-oss-120b [20] under a visual-effect-equivalence rubric that is tolerant to paraphrases; for example, “thicker text” credits a font_weight \(400\!\to\!700\) mutation. The three judge LLMs (gpt-oss-120b, Kimi K2.5, Qwen3.5-VL-397B) reach mean pairwise Cohen \(\kappa = 0.93\) and produce identical 13-model rankings (§3.5); the full prompt is in §12.2.

3.2 Main Results↩︎

0pt 0pt 6pt

@ >

X c | rrr r | r >cr & & & &
& & Easy & Med & Hard & Overall & &

Kimi K2.5 & 1T / 32B & 54.2 & 36.4 & 18.6 & 36.4 & 87.2 & 42.2
Qwen3.5-VL-397B & 397B / 17B & 45.1 & 31.5 & 13.7 & 30.1 & 96.6 & 37.6
Qwen3-VL-235B-Thinking & 235B / 22B & 30.1 & 17.3 & 10.5 & 19.3 & 98.8 & 28.3
GLM-4.6V-Flash & 9B & 24.5 & 17.6 & 9.3 & 17.1 & 75.8 & 23.8
GLM-4.6V & 106B / 12B & 17.0 & 10.9 & 5.5 & 11.2 & 99.6 & 21.2
Qwen3-VL-30B-Instruct & 30B / 3B & 14.5 & 9.0 & 4.5 & 9.3 & 82.0 & 17.6
Qwen3-VL-30B-Thinking & 30B / 3B & 16.5 & 8.8 & 3.8 & 9.7 & 77.8 & 17.5
Qwen3-VL-235B-Instruct & 235B / 22B & 9.6 & 3.0 & 2.6 & 5.1 & 100.0 & 15.9
InternVL3.5-30B-A3B & 30B / 3B & 4.7 & 3.9 & 3.8 & 4.2 & 100.0 & 15.0

Gemini 3.1 Pro & — & 60.5 & 38.9 & 22.7 & 40.7 & 98.4 & 47.2
Gemini 3 Flash & — & 52.5 & 32.5 & 18.2 & 34.4 & 91.4 & 40.9
Claude Opus 4.7 & — & 41.2 & 30.5 & 21.8 & 31.2 & 99.6 & 38.9
GPT-5.4 & — & 48.8 & 30.5 & 12.2 & 30.5 & 99.6 & 38.3

Table [tab:main95table] reports Visual Diff Detection performance for all 13 models on the full 4,400-pair benchmark.

A low ceiling on true-change detection. Gemini 3.1 Pro leads the leaderboard at 47.2% Accuracy, 5.0 pp ahead of Kimi K2.5 (42.2%) and 32.2 pp above InternVL3.5 (15.0%). Yet this aggregate score masks a sharper failure on the has-diff slice: even the leader identifies only 40.7% of ground-truth mutations, missing roughly three of every five true visual changes. Seven of thirteen models fall below 30% Accuracy, and two models (Qwen3-VL-235B-Instruct and InternVL3.5) clear the trivial always-no-diff baseline of 11.4% Accuracy by less than 5 pp. Open-ended Visual Diff Detection on real web-UI pairs is therefore far from solved.

Hard mutations remain difficult for every model. Recall drops sharply from Easy to Hard across the strongest non-abstaining models: Gemini 3.1 Pro falls from 60.5% to 22.7% (\(-\)37.8 pp), GPT-5.4 from 48.8% to 12.2% (\(-\)36.6 pp), Kimi K2.5 from 54.2% to 18.6% (\(-\)35.6 pp), and Gemini 3 Flash from 52.5% to 18.2% (\(-\)34.3 pp). Hard-tier Recall stays below 23% for every model, indicating that the hardest cells expose a substantially sharper perception failure rather than merely a weaker version of the Easy setting.

No-diff pairs separate sensitivity from restraint. The no-diff slice reveals that higher Recall can come with hallucinated differences. Aggressive reporters hallucinate frequently: GLM-4.6V-Flash marks 24.2% of no-diff pairs as changed, and the Qwen3-VL 30B variants hallucinate on 18–22% of no-diff pairs, reducing their Accuracy relative to has-diff Recall. At the other extreme, Claude Opus 4.7, GPT-5.4, and GLM-4.6V hallucinate on only 0.4% of no-diff pairs; Qwen3-VL-235B-Instruct and InternVL3.5 reach 100.0% No-Diff specificity largely by reporting almost nothing on either changed or unchanged inputs. Thus, no-diff controls are necessary to distinguish genuine visual sensitivity from either over-reporting or abstention.

Reasoning helps only at larger scale. The four Qwen3-VL variants form a 2\(\times\)2 grid over {30B, 235B} \(\times\) . At 30B, switching to Thinking leaves Accuracy essentially unchanged (\(17.6\,\to\,17.5\), \(-\)0.1 pp). At 235B, the same switch improves Accuracy by 12.4 pp (\(15.9\,\to\,28.3\)), driven mainly by a 20.5 pp gain in Easy Recall (\(9.6\,\to\,30.1\)). Within the Instruct setting, scaling from 30B to 235B does not improve performance (\(17.6\,\to\,15.9\)), suggesting that size alone is insufficient; in this model family, reasoning mode becomes useful only at larger scale.

3.2.0.1 Sensitivity–restraint trade-off.

Plotting has-diff Recall against no-diff hallucination rate (Appendix 8) reveals a Pareto frontier reached only by the three closed-source frontier APIs: Gemini 3.1 Pro, Claude Opus 4.7, and GPT-5.4. No open-weight model enters the “accurate and restrained” region (Recall \(\geq30\%\), hallucination \(\leq5\%\)), showing that current models must jointly improve sensitivity to real changes and restraint on unchanged pairs.

3.3 Property-Level Failure Modes↩︎

Figure 3: **Recall heatmaps across 13 models.** Cells: has-diff Recall (%). Columns: models, sorted by overall Recall (best at left). (a) Rows: 13 CSS operators (300 has-diff pairs each). (b) Rows: 15 source-domain labels (§2). The operator panel varies sharply by row while the domain panel is nearly flat: capability is property-specific, not domain-specific..

Figure 3 compares recall along two axes: CSS operators and source domains. The operator heatmap (Figure 3 (a)) shows large, structured differences across visual properties. Some operators are consistently visible: justify, which moves whole rows of text, reaches 87.0% Recall with Gemini 3.1 Pro; text substitutions can often be read through OCR, with Claude Opus 4.7 reaching 70.3%; and opacity reaches 58.3% with Kimi K2.5. In contrast, several operators remain difficult for nearly all models: gradient reaches only 26.7% at best, line_height has a median Recall of 4.0%, and rounded has a median Recall of 13.3%. The top-performing model also changes by operator: the gold top-1 outlines split across Gemini 3.1 Pro, Claude Opus 4.7, Kimi K2.5, and Qwen3.5-VL-397B. Thus, an aggregate leader is not uniformly best across visual properties.

The domain heatmap (Figure 3 (b)) shows the opposite pattern. Recall varies much less across the 15 source-domain labels assigned during construction. The easiest domain (portfolio, mean Recall 25.3%) and the hardest (entertainment, mean Recall 18.2%) differ by only 7.1 pp, while the best–worst model spread is 32.2 pp. Model rankings are also stable across domains: Gemini 3.1 Pro is top-1 on all 15 domain rows, and the lowest-performing models remain at the bottom regardless of domain. Together, Figure 3 shows that the capability gap is property-specific rather than domain-specific: what changed matters more than where it appeared.

3.4 Visual Magnitude Does Not Explain Difficulty↩︎

Figure 4 tests whether operator difficulty can be explained by the size of the visual signal. Across the 13 CSS operators, bbox-level pixel change is essentially uncorrelated with mean Recall (\(r=-0.08\), \(r^2<1\%\); Figure 4 (a)). Operators such as text and position occupy the low-pixel-change region but achieve high Recall, while gradient and line_height produce larger pixel changes yet remain among the hardest operators. Thus, larger pixel differences are not necessarily easier for VLMs to detect.

CLIP image-embedding distance does not explain difficulty either (\(r=+0.06\), \(r^2<1\%\); Figure 4 (b)). The CLIP distances for all operators are compressed into a narrow range, suggesting that caption-aligned image features are themselves insensitive to many CSS-attribute-level mutations. The two views also fail in different ways: gradient has large pixel change but relatively small CLIP distance, while letter_spacing has high CLIP distance because character positions shift across a paragraph, even though model Recall remains modest. These patterns suggest that DiffSpot difficulty is not governed by visual magnitude alone, but by whether a model can perceive and name the changed CSS-level visual property.

3.5 Robustness to Judge Choice↩︎

Because DiffSpot evaluates open-ended model responses, we verify that the leaderboard is not an artifact of a single judge LLM. We re-score the full 13-model \(\times\) 4,400-case grid with two additional judges (Kimi K2.5 and Qwen3.5-VL-397B) using the same prompt and visual-effect-equivalence rubric as gpt-oss-120b. Across the three judge pairings, per-case agreement is high (Cohen’s \(\kappa=0.92\)–\(0.94\)), and the 13-model ranking is unchanged (Kendall’s \(\tau=1.00\) for every pair, computed on the full grid). The residual mean Accuracy differences (\(\leq\!1.6\) pp in any pair) are small calibration shifts rather than leaderboard reshufflings. We therefore report main-table numbers under gpt-oss-120b, the strictest judge; all comparative claims hold under all three judges.

3.6 Realness Audit by Independent VLM Judges↩︎

We compare each original webpage screenshot against its code-rendered counterpart by asking three VLM judges (Qwen3.5-VL-397B, Kimi K2.5, Gemini 3 Flash) to rate every image on a 1–5 realness scale in isolation, with no cue to which image is which. Pairs are grouped by content richness into rich, standard, and minimal, and Table 1 reports the paired mean difference \(\Delta = \mu(\text{orig}) - \mu(\text{rend})\) within each group. All three judges rate originals and renderings within 0.3 points on a 5-point scale, with renderings scoring marginally higher across every group; the per-pair differences across judges are essentially uncorrelated.

Table 1: Paired realness \(\Delta = \mu(\text{orig}) - \mu(\text{rend})\)on a 1–5 scale, stratified by visual style.
VLM judge	rich	standard	minimal	Overall
Qwen3.5-VL-397B	\(-0.15\)	\(-0.22\)	\(-0.34\)	\(\mathbf{-0.23}\)
Kimi K2.5	\(-0.14\)	\(-0.22\)	\(-0.04\)	\(\mathbf{-0.13}\)
Gemini 3 Flash	\(-0.16\)	\(-0.34\)	\(-0.40\)	\(\mathbf{-0.30}\)

4 Related Work↩︎

DiffSpot is a spot-the-difference benchmark for fine-grained visual perception on rendered web interfaces. We position it against two closely related lines of work: fine-grained visual perception benchmarks (§4.1) and visual-difference benchmarks (§4.2).

4.1 Fine-Grained Visual Perception Benchmarks↩︎

A growing line of benchmarks probes whether VLMs perceive fine-grained visual content beyond what high-level VQA requires. Single-image perception probes reformat classic CV tasks as VQA, MCQ, or yes/no judgement [1], [26]–[30]; image-pair and multi-image probes formulate comparative perception as MCQ, yes/no, or paired retrieval [31]–[35]; and a parallel line studies object-presence, attribute, and relational hallucination on single images via yes/no polling, open-ended VQA, or caption-based CHAIR analysis [36]–[39]. Other fine-grained perception benchmarks [40]–[42] cover adjacent domains.

These benchmarks establish that VLMs can struggle with fine-grained visual details, but they differ from DiffSpot in task form and control. Most evaluate single-image perception or closed-form comparative judgments, and each item is typically treated as a single difficulty point. DiffSpot instead takes paired screenshots as input, elicits an open-ended list of visual differences, and places every has-diff item on a controlled per-property \(\times\) per-magnitude grid.

4.2 Visual Difference Benchmarks↩︎

The most direct comparison is to image-pair visual-difference benchmarks. OmniDiff [5] provides a large human-annotated image-difference captioning dataset and evaluates generated captions with overlap metrics such as BLEU, METEOR, ROUGE, CIDEr, and SPICE, which reward paraphrastic similarity to a reference caption rather than structural coverage of a change list. VLM-SubtleBench [6] studies subtle paired-image differences across multiple domains, but is primarily evaluated through multiple-choice questions, with only a subset using captioning-style evaluation.

A pre-VLM change-captioning lineage also studies visual differences, typically pairing each dataset with a specialized captioning or contrastive model [43]–[50]. Other image-pair benchmarks target adjacent capabilities, including image-set differences, generalist diff-captioning, medical difference VQA, and image-manipulation description [51]–[53]. A complementary line evaluates the generation side of image editing, scoring whether a generative model can reproduce a target edit given a known before/after pair [54]–[56]; DiffSpot instead evaluates the perception-side problem of identifying what changed.

DiffSpot differs from these benchmarks along five axes. First, domain focuses on rendered web interfaces rather than natural images or generic image pairs, making it possible to specify visual changes through the underlying HTML/CSS. Second, ground truth is fully programmatic: each label is derived from a CSS mutation record rather than human or LLM-generated annotation. Third, stratification places each has-diff pair on a per-property \(\times\) per-magnitude grid (13 operators \(\times\) 3 difficulty tiers, 100 pairs per cell). Fourth, evaluation elicits an open-ended diff list on every item and structurally matches each response against the mutation record. Fifth, hallucination control pairs the has-diff slice with 500 no-diff pairs, measuring false-positive difference reports under the same protocol.

5 Conclusion↩︎

DiffSpot evaluates whether VLMs can identify fine-grained visual differences on rendered web interfaces. By constructing pairs through programmatic HTML/CSS mutation, it provides machine-readable ground truth, controlled property–magnitude stratification, and open-ended diff-list evaluation without caption-overlap or self-retrieval surrogates. Across 13 recent and frontier VLMs, the task remains far from solved: even the strongest model identifies only 40.7% of true visual changes, and Hard-tier Recall stays below 23% for every model. DiffSpot further shows that failures are property-specific rather than domain-specific, that pixel and CLIP magnitudes do not explain difficulty, and that no-diff pairs expose a sensitivity–restraint trade-off. The benchmark, evaluation harness, and self-contained HTML regeneration pipeline are released alongside the paper.

6 Limitations and Broader Impacts↩︎

6.0.0.1 Limitations.

DiffSpot targets rendered web interfaces; extending the property-level evaluation paradigm to mobile UI and desktop application screenshots is left to future work. Source pages are predominantly English, so multilingual and right-to-left layouts are a natural extension. The no-diff metric counts whether a model fabricates any change; localizing the specific false-positive claim is left to future work.

6.0.0.2 Broader impacts.

DiffSpot supports research on fine-grained visual change detection for web interfaces, with practical relevance to UI regression testing and accessibility-oriented quality assurance. Better models and evaluation tools in this area could reduce manual QA effort and make semantic UI testing more accessible to smaller teams. As with other web automation benchmarks, misuse is possible if similar techniques are applied to monitor public webpages at scale. We mitigate data-release risks by filtering candidate pages for personally identifiable information and adult content, and by releasing only rendered screenshots of pages that were publicly accessible at collection time.

7 Visual-Reviewer Audit of the LLM Judge↩︎

Throughout the main paper we adjudicate VLM responses against the ground-truth mutation using an LLM judge (gpt-oss-120b). The cross-LLM-judge ranking-stability analysis is in §3.5; this appendix documents a complementary check that compares the LLM judge’s verdicts against image-aware adjudication on a random sample.

7.0.0.1 Setup.

We draw a seed-42 random sample of \(n{=}477\) unique judge verdicts. Three independent volunteer reviewers—labelled Reviewer A, Reviewer B, and Reviewer C—are each shown, for every sampled case: (i) the before/after screenshots, (ii) the VLM’s free-form answer, (iii) the ground-truth mutation description, and (iv) the judge’s verdict together with its reasoning. Each reviewer returns an audit verdict in \(\{\texttt{correct},\texttt{wrong}\}\): correct means the reviewer, after looking at the images, agrees that the judge’s decision is consistent with what the VLM actually said versus what was actually changed; wrong means the reviewer believes the judge has made an error visible from the image. Reviewers work independently; they do not see each other’s verdicts. The audit was a small voluntary expert sanity check rather than a crowdsourcing or paid annotation study. The reviewers were not used to construct the dataset or determine the benchmark ground truth; they only inspected a random sample of LLM-judge decisions to validate judge reliability.

Table 2: Rate at which each independent visual reviewer agrees with thejudge’s verdict, broken down by task type. The majority rowreports the fraction of cases on which at least two of three reviewersagreed with the judge.
Audit reviewer	has_diff	no_diff	Overall
Reviewer A	\(85.6\%\)	\(98.3\%\)	\(87.2\%\)
Reviewer B	\(95.0\%\)	\(98.3\%\)	\(95.4\%\)
Reviewer C	\(96.6\%\)	\(100.0\%\)	\(97.1\%\)
Majority (\(\geq\)2/3)	\(\mathbf{96.9\%}\)	\(\mathbf{98.3\%}\)	\(\mathbf{97.1\%}\)

7.0.0.2 Analysis.

All three reviewers independently agree with the judge on at least \(87\%\) of the 477 audited cases; two of the three agree above \(95\%\). Taking the majority vote across reviewers as a best-available proxy for a well-calibrated human gold standard, the judge’s verdicts are upheld on \(97.1\%\) of cases overall—\(96.9\%\) on has_diff and \(98.3\%\) on no_diff. Inter-reviewer agreement is high: all three reviewers reach the same verdict on \(84.7\%\) of cases. Crucially, the fraction of cases that all three reviewers unanimously flag as a judge error is only \(5/477 = 1.0\%\), which we take as the strongest empirically supported upper bound on the judge’s error rate on this sample. The remaining \(\sim\)14% of cases where reviewers split are overwhelmingly borderline judgments—partial matches, paraphrases with ambiguous scope, or mutations on elements whose visual manifestation is subtle—rather than clear-cut errors.

7.0.0.3 Takeaway.

The LLM judge’s decisions are almost entirely consistent with what image-aware reviewers conclude on the same cases. Both the relative model ranking and the absolute Accuracy numbers reported in the paper are therefore robust to the choice of judging procedure. The 1% three-way-unanimous error rate provides a tight bound for readers who wish to reason about residual noise in our evaluation.

8 Accuracy–Hallucination Trade-off↩︎

The aggregate Accuracy in Table [tab:main95table] combines two behavioural axes—has-diff Recall and no-diff specificity—into a single score. A two-dimensional view exposes failure modes that a one-dimensional ranking hides.

Figure 6 plots each model’s overall has-diff Recall against its no-diff hallucination rate—the fraction of 500 pixel-identical pairs on which the model still reports a change. The 2D view exposes three distinct failure modes. (i) Three closed-source frontier APIs land in the ideal zone (Recall \(\geq\) 30%, hallucination \(\leq\) 5%): Gemini 3.1 Pro (Recall 40.7%, halluc.%), Claude Opus 4.7 (31.2%, 0.4%), and GPT-5.4 (30.5%, 0.4%) all achieve high recall of real changes and discipline when the input is unchanged. No open-weight or privately-deployed model reaches this region. (ii) Three open-weight variants—GLM-4.6V-Flash (Recall 17.1%, halluc.%), Qwen3-VL-30B-Thinking (9.7%, 22.2%), and Qwen3-VL-30B-Instruct (9.3%, 18.0%)—are simultaneously weak and trigger-happy, the worst cost/benefit combination; one in five to one in four of their no-diff outputs is a fabrication. (iii) Models at halluc.\(=\) 0.0% (InternVL3.5-30B-A3B and Qwen3-VL-235B-Instruct) do not exhibit principled selectivity: their has-diff Recall of 4.2% and 5.1% shows that they report almost nothing on any input, so they never hallucinate because they never speak. Kimi K2.5 and Gemini 3 Flash occupy the high-Recall middle band but purchase their Recall with 9–13% hallucination, while Claude Opus 4.7 (halluc.%, Recall 31.2%) and GPT-5.4 (0.4%, 30.5%) sit at the opposite extreme as the two “conservative but competent” operating points; Claude marginally dominates GPT-5.4 on this plane (same hallucination rate, 0.7 pp higher Recall). The Pareto frontier reduces to the segment from Claude Opus 4.7 to Gemini 3.1 Pro; every other model is dominated, including GPT-5.4.

Figure 6: Has-diff Recall vs.no-diff hallucination rate. Each point is one of the 13 evaluated VLMs. The y-axis is overall Recall on the 3,900 has-diff pairs; the x-axis is the hallucination rate on 500 no-diff pairs (fraction reporting any change). Green: the three models in the “accurate and restrained” region (Gemini 3.1 Pro, Claude Opus 4.7, and GPT-5.4). Red: three models that combine weak Recall with high hallucination. Blue: the remaining seven. Models at halluc.= 0.0% (InternVL3.5-30B-A3B, Qwen3-VL-235B-Instruct) reach that position by producing near-empty outputs rather than by genuine selectivity. — Figure 6: **Has-diff Recall vs.no-diff hallucination rate.** Each point is one of the 13 evaluated VLMs. The \(y\)-axis is overall Recall on the 3,900 has-diff pairs; the \(x\)-axis is the hallucination rate on 500 no-diff pairs (fraction reporting any change). Green: the three models in the “accurate and restrained” region (Gemini 3.1 Pro, Claude Opus 4.7, and GPT-5.4). Red: three models that combine weak Recall with high hallucination. Blue: the remaining seven. Models at halluc.\(=\) 0.0% (InternVL3.5-30B-A3B, Qwen3-VL-235B-Instruct) reach that position by producing near-empty outputs rather than by genuine selectivity.

9 Benchmark Size Justification↩︎

DiffSpot fixes 100 records per (operator \(\times\) difficulty) cell, giving 3,900 has-diff records (13 operators \(\times\) 3 difficulties \(\times\) 100). We empirically verify that this size is sufficient to produce a stable 13-model ranking using stratified sub-sampling.

9.0.0.1 Setup.

For each \(K \in \{10, 20, \ldots, 100\}\) records-per-cell, we draw \(200\) stratified random subsamples (each cell sampled independently), recompute the 13-model has-diff Recall ranking on each subsample, and measure Kendall’s \(\tau\) versus the full \(K=100\) ranking. The stratification fixes the operator/difficulty composition, so the only varying factor is sample size. Top-1 model preservation and full ranking exact-match rates are tracked alongside.

9.0.0.2 Convergence.

Figure 7 plots mean \(\tau\) with the 95% confidence interval over the 200 reps. The lower bound of the 95% CI first reaches \(\geq 0.95\) at \(K = 80\) per cell (\(N = 3{,}120\)) and \(\geq 0.99\) at \(K = 100\) per cell (\(N = 3{,}900\), the paper setting). The top-1 model (Gemini 3.1 Pro) is preserved in \(\geq 99.5\%\) of subsamples for every \(K \geq 10\) and in \(100\%\) for every \(K \geq 20\). The benchmark size we report sits exactly at the operating point where ranking stability saturates: smaller \(K\) produces visibly wider CI bands, while \(K > 100\) would not be reachable without enlarging the source pool.

Figure 7: Stratified per-cell ranking stability. Mean Kendall’s \tau between the 13-model has-diff Recall ranking on a random subsample (drawing K records per (operator \times difficulty) cell, 39 cells, N = 39 K) and the full ranking (K = 100, N = 3{,}900). Shaded band: 95% CI over 200 random subsamples per K. Dashed lines mark \tau = 0.95 and \tau = 0.99. The 95% CI lower bound first reaches 0.95 at K = 80 and 0.99 at K = 100. — Figure 7: **Stratified per-cell ranking stability.** Mean Kendall’s \(\tau\) between the 13-model has-diff Recall ranking on a random subsample (drawing \(K\) records per (operator \(\times\) difficulty) cell, 39 cells, \(N = 39 K\)) and the full ranking (\(K = 100\), \(N = 3{,}900\)). Shaded band: 95% CI over 200 random subsamples per \(K\). Dashed lines mark \(\tau = 0.95\) and \(\tau = 0.99\). The 95% CI lower bound first reaches \(0.95\) at \(K = 80\) and \(0.99\) at \(K = 100\).

10 Mutation Mechanics↩︎

The mutator selects one of two mechanisms per operator:

10.0.0.1 (i) Tailwind CSS class swap.

For operators with a natural class taxonomy (e.g. rounded-lg\(\to\)rounded-none, bg-blue-700\(\to\)bg-blue-100), the mutator parses the target element’s class attribute with exact fullmatch (no prefix-leak), skips responsive variants (e.g.hover:, sm:), and swaps in the new class.

10.0.0.2 (ii) Inline `!important` style override.

For letter_spacing and line_height—where Tailwind’s discrete scale does not span a range wide enough to produce reliably non-trivial pixel deltas—the mutator appends style="letter-spacing: 0.12em !important" (or analogous) to the target element.

Both paths operate on static HTML before re-rendering.

10.0.0.3 Per-operator parameter ranges.

Step-based operators (rounded, color, font_weight, font_size, border, opacity, justify, position, spacing, gradient) use a max_steps parameter bounding Tailwind-scale distance between before and after values: Easy \(=\) 3–5 steps, Medium \(=\) 2, Hard \(=\) 1. Continuous-valued operators (letter_spacing, line_height) use em-offset magnitude: Easy \(=\) \(\pm0.20\), Medium \(=\) \(\pm0.12\), Hard \(=\) \(\pm0.06\). The text operator uses character-substitution count (Easy \(=\) 5+, Medium \(=\) 2–4, Hard \(=\) 1).

11 Construction Pipeline Details↩︎

11.0.0.1 CLIP-similarity gate threshold.

The \(\geq 0.70\) threshold was set via pilot inspection of \(\sim\)200 borderline pairs in the \([0.65, 0.75]\) band. Pairs above 0.70 preserved layout structure and font choices; pairs below began to show layout drift.

11.0.0.2 Domain/style labeling.

After PII / dynamic-tag / abnormal-length filters, an LLM labeler (gpt-oss-120b [20] at temperature 0.3, with a fixed prompt enumerating 15 domain categories and 4 visual-style categories) assigns each page a domain and style tag. A capped-natural sampling policy of at most 800 pages per domain bounds the largest domain to \(\sim\)7% of the corpus, yielding the multi-label enriched pool reported in Figure 1 (Panel A).

11.0.0.3 Four-stage content filter (475 records removed).

Text-truncation (96): the text_swap operator truncates element text to 50 characters for its diff record; for long strings whose diff position falls beyond character 50, the recorded old and new collapse to the same prefix.
Bbox-failure (190): mutations that push the target out of the viewport make getBoundingClientRect() return a null or clipped rectangle, breaking the outside-bbox locality check.
Font-rendering tofu (30): headless Chromium lacks Noto Sans Korean / Thai / Devanagari, so pages containing those scripts render as empty glyph boxes.
Viewport overflow (168): if the target element extends more than 30% below the 800-pixel viewport bottom, the visible portion of the change is too partial for fair comparison.

Filters are applied in union; after dedup of overlap, 475 records are removed, leaving 20,629 filtered candidates.

11.0.0.4 Cell-balanced sampling rationale.

We pick \(n=100\) per cell because the per-cell binomial standard error at \(p=0.5\) is \(\approx 5\) pp—tight enough for per-cell reporting while staying within the supply of the scarcest operator-tier cell after filtering.

12 Prompts Used in Evaluation↩︎

For full reproducibility we reproduce the prompts used for both the VLM evaluation step (§12.1) and the LLM judge step (§12.2). Verbatim text is shown exactly as passed to the models at inference time.

12.1 VLM Prompt↩︎

Every evaluated VLM receives the two rendered screenshots (before / after) together with the user prompt shown below. The wording uses the permissive “a change may have been made” form rather than asserting that a change has occurred. Asserting a change primes the model on no-diff inputs and inflates the hallucination rate; the permissive phrasing leaves both has-diff and no-diff records as first-class cases.

VLM User Prompt Look at these two webpage screenshots carefully (Image 1 = before, Image 2 = after). A change may have been made between the two screenshots.

Your task: Find and describe ALL visual differences you can spot. - Be specific about the location (e.g., "the heading at the top", "the button in the sidebar") - Describe what changed and how (e.g., "the text became bolder", "the spacing increased") - Don’t skip any difference, no matter how small - IMPORTANT: Only report differences you can actually SEE in the images. Do NOT guess or make up differences. - If you cannot find any difference, say "No visible differences found."

List each difference on a separate line starting with "- ".

12.2 Judge Prompt↩︎

The judge prompt is a templated document composed of (i) a fixed base template injecting the ground-truth mutation, the VLM’s free-form answer, and general element-matching rules; and (ii) one operator-specific rule block injected at {operator_rule} based on the ground-truth mutation type (see §12.3). For no-diff cases the operator slot is replaced by a no-diff rule. Per-call prompt length is \({\sim}\)1,100–1,350 tokens.

Judge Prompt: Base Template You are evaluating whether a Vision-Language Model (VLM) response describes a specific webpage change. You will compare the VLM’s text against either a single ground-truth change, or no change at all.

## Ground Truth Change gt_json

## Ground Truth Description (reference only) gt_answer

## VLM Response The VLM was shown both screenshots and asked to find all visual differences. Here is the VLM’s response: — vlm_response —

## General element matching

- ‘header’ "top nav" / "top bar"; ‘hero section’ "banner"; ‘footer’ "bottom section"; ‘button’ "CTA"; ‘card’ "tile" / "panel"; ‘heading’ "headline" / "h1/h2". - Container <-> child: when GT says "section X changed" and VLM describes children of X all showing the change -> match (effects like opacity propagate to children). - Text matched by quoted content: when VLM quotes the SAME before/after text as GT – match, regardless of how VLM labels the section. - Quoted GT text is a prefix: the quoted text inside the GT description identifies the element by its opening characters, not the full string. A VLM quote that starts with the same characters and continues further refers to the same element. - Distinctive text token overrides location labeling: If both GT and VLM reference a distinctive text token (button label, menu item, author name, etc.), they refer to the SAME element even if VLM labels its surrounding region differently from GT.

## PRINCIPLE OVER KEYWORDS

The operator-specific rule below has an ESSENCE (the physical/visual reality) and ACCEPT/REJECT criteria. Apply the principle, not keyword matching. The ACCEPT examples are illustrative – any VLM phrasing that semantically matches the principle should be accepted, even if wording differs from examples. REJECT only when the description genuinely contradicts the physical effect.

## Operator-specific Rule

operator_rule

## Rules of thumb

- Direction consistency required for all non-opacity operators. - Color-name precision NOT required if direction matches. - Extra claims by VLM = hallucinations, NOT verdict-wrong: if VLM correctly paraphrases the GT change AND adds extra unrelated claims, verdict stays correct; the extras go to ‘hallucinations’. - Conclude "wrong" ONLY after applying the operator-specific principle and confirming no semantic match.

## Your Task

Step 1: Reasoning. Write out: 1. What does the GT change describe (element, property, direction)? 2. Which parts of the VLM response could refer to this change? Quote the relevant text. 3. Apply the operator-specific principle above. Determine if the VLM’s description matches the ESSENCE or ACCEPT criteria. If no match, check whether it falls under REJECT or is simply unrelated.

Step 2: Verdict. Assign one of: - correct: VLM identifies the GT element specifically AND describes a change matching the operator’s ACCEPT criteria. - wrong: VLM did not mention the element specifically; OR mentioned the element but did not claim any change; OR described a different property / reversed direction failing the operator’s REJECT condition.

## Hallucinations

After the verdict, list every VLM-reported difference that does NOT correspond to the GT change. Each distinct extra claim is a separate entry. If GT is null, every specific VLM-reported difference is a hallucination.

## Output

Return ONLY valid JSON (no markdown fences, no commentary):

"reasoning": "step 1 analysis with operator-specific principle check", "gt_verdict": "correct" | "wrong" | null, "vlm_match": "quote from VLM that matches the GT change, or empty", "hallucinations": ["each distinct hallucinated difference"], "summary": "correct": 0, "wrong": 0, "hallucinated": 0

If GT is null, set ‘gt_verdict’ to null, ‘vlm_match’ to empty, correct/wrong to 0.

Judge Prompt: No-diff Rule (injected when GT is null) **This case: NO DIFFERENCE** (the two screenshots are identical)

There is no GT change. The verdict is null. Any specific visual difference reported by the VLM is a hallucination.

ACCEPT (verdict null with empty halluc list): - VLM correctly says no differences ("no visible differences found", "the two images appear identical", etc.)

REJECT (verdict null but each VLM claim becomes a hallucination): - VLM reports any specific change (color, position, spacing, text, etc.) – list each as a separate hallucination

12.3 Per-Operator Rules↩︎

For has-diff cases the judge receives one of 13 operator-specific rule blocks, selected by the ground-truth mutation type. Every rule follows the same four-part structure: (1) a one-sentence PRINCIPLE describing the physical/visual signature of the operator; (2) an ACCEPT section listing paraphrases and anchor phrasings; (3) a REJECT section specifying what counts as a genuine contradiction; (4) confirmed anchor phrasings observed across models. Table 3 summarises the thirteen principles.

Table 3: One-line principle for each of the 13 per-operator judgerules. The full ACCEPT/REJECT text is released with the evaluationcode; two illustrative full rules (opacity,spacing) are reproduced below.
Operator	Principle
opacity	Opacity decrease blends the element’s colour toward the background; direction (lighter/darker) depends on background luminance.
position	Element moves \(N\) pixels in the GT direction; adjacent layout (gaps, overlaps, line wraps, separators) changes as a consequence.
spacing	Container padding / gap changes; contents shift away from the padded edge when padding increases, toward it when it decreases.
justify	Container’s child items redistribute along the main axis; any individual child’s shift is valid evidence of the redistribution.
letter_spacing	Gap between characters changes; text (and containing button/label) becomes wider or narrower.
font_weight	Character stroke thickness changes; bolder = thicker, lighter = thinner.
font_size	Character size changes; text block occupies more/less space and may wrap to more/fewer lines.
line_height	Vertical spacing between lines of a paragraph changes (looser / tighter).
color	Colour moves to a darker or lighter shade within the same hue family; colour-name precision is tolerant.
border	Colour of the visible outline changes; “border”, “line”, “outline” are treated as synonyms.
rounded	Corner shape changes (sharp \(\leftrightarrow\) rounded/pill/circular).
gradient	Axis of colour transition changes (horizontal \(\leftrightarrow\) vertical \(\leftrightarrow\) diagonal, or reversal of the same axis); positional colours rearrange.
text	Specific characters / words / punctuation are altered; the exact before\(\to\)after substitution is the identifier, region labelling secondary.

Judge Prompt: Operator Rule for opacity **This case: OPACITY decrease** (e.g., 100 PRINCIPLE: When opacity decreases, the element’s color blends toward the background color (semi-transparent overlay). Direction depends on bg: dark element on light bg -> appears LIGHTER; light element on dark bg -> appears DARKER. Both directions valid as long as the new color is closer to surrounding bg.

Key paraphrase equivalence: The visual signature of opacity reduction IS a color shift on the same element. When VLM correctly identifies the GT element AND describes any perceptible color change on it (shade shift, "became lighter", "became faded", "changed to lighter blue", "pale version of the original color"), this is a valid description – even if VLM does not use the word "opacity" or "transparent".

CRITICAL – Judge cannot see images, so cannot verify bg color directly. For opacity DECREASE, ACCEPT EITHER direction (lighter OR darker) when: - VLM describes consistent direction across multiple sub-elements (e.g., "text + button + placeholder all became darker") -> strong evidence of uniform opacity blend toward bg - VLM uses opacity-language ("faded / ghosted / transparent / semi-transparent") in EITHER direction -> ACCEPT regardless of light/dark bg assumption - Direction-only descriptions ("became darker" / "became lighter") on the GT element -> ACCEPT (Judge cannot verify which direction is "right" without bg)

REJECT direction ONLY when: - VLM says "more vivid / more saturated / brighter and bolder / sharper contrast" -> clearly OPPOSITE effect (increased opacity) - VLM describes hue shift unrelated to bg blend (e.g., "red -> blue" on a white bg)

Judge Prompt: Operator Rule for spacing **This case: SPACING/PADDING change** (container padding/margin/gap increased or decreased)

PRINCIPLE: Container’s inner space changes. Contents physically shift AWAY from the padded edge when padding INCREASES, and TOWARD the edge when padding DECREASES. For gap changes (between sibling items), gap visibly grows/shrinks.

No CSS terminology required: VLMs see visual consequences, not CSS properties. ACCEPT descriptions of visual consequences WITHOUT requiring the VLM to name "padding" / "margin" / "gap" / any CSS keyword. Valid phrasings include: - "the box / container became wider / narrower / taller / shorter" - "there is more / less empty space between X and Y" - "X is closer to / further from Y" - "the content shifted inward / outward / got pushed to the right" - "elements are more spread out / more clustered / closer together" - "text has more / less room around it"

ACCEPT any description matching this principle: - Content shift (consistent direction): left padding UP -> "contents pushed right"; vertical padding UP -> "X moved downward" / "more space above" - Container resize: "box visibly wider/taller" (for UP) or "narrower/shorter / more compact" (for DOWN) - Empty-space descriptions: "there is more empty space between the left border of the box and the content" – ACCEPT even without saying "padding" - Internal element changes: "input fields are wider" (when container padding DOWN frees internal space) - Text wrap consequence: "text wraps to more lines" / "fewer lines" - Gap changes: "horizontal/vertical spacing between X and Y decreased/increased"

REJECT only when: - Direction genuinely contradicts (padding UP described as "content shifted toward edge / less space" is opposite) - VLM describes a completely unrelated element that is not in or adjacent to the GT container

The remaining eleven per-operator rules follow the same ESSENCE / ACCEPT / REJECT / anchor-phrasings structure, with rule-specific examples tuned to each operator’s visual signature. The full text is released with the evaluation code (scripts/09_judge_single_model.py, function get_operator_rule).

13 Compute Resources↩︎

The full evaluation grid comprises \(13 \times 4{,}400 = 57{,}200\) open-ended VLM generations on the benchmark, scored by three independent judge LLMs for a total of \(\sim 1.7 \times 10^{5}\) judge verdicts. Closed-source frontier models (Anthropic, OpenAI, Google families) are queried through their public APIs at default decoding temperatures. Open-weight VLMs and the judge LLMs are served on NVIDIA H20 GPU pods. We do not report wall-clock figures because end-to-end runtime is dominated by external API rate limits rather than local compute, but the H20 footprint is sized to comfortably hold the largest evaluated open-weight model (a 235B-parameter MoE) alongside the gpt-oss-120b judge, and the full grid was reproduced more than once during pipeline development.

References↩︎

[1]

S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie, “Eyes wide shut? Exploring the visual shortcomings of multimodal LLMs,” in CVPR, 2024, doi: 10.1109/CVPR52733.2024.00914.

[2]

T. Luo, A. Cao, G. Lee, J. Johnson, and H. Lee, “Probing visual language priors in VLMs,” in Proceedings of the 42nd international conference on machine learning (ICML), 2025.

[3]

H. S. Shahgir et al., “VLMs need words: Vision language models ignore visual detail in favor of semantic anchors,” arXiv preprint arXiv:2604.02486, 2026, [Online]. Available: https://arxiv.org/abs/2604.02486.

[4]

Anthropic, “Claude models overview.” https://docs.anthropic.com/en/docs/models-overview, 2026.

[5]

Y. Liu, S. Hou, S. Hou, J. Du, S. Meng, and Y. Huang, “OmniDiff: A comprehensive benchmark for fine-grained image difference captioning,” arXiv preprint arXiv:2503.11093, 2025.

[6]

M. Kim, S. Lee, and D. Park, arXiv:2603.07888“VLM-SubtleBench: How far are VLMs from human-level subtle comparative reasoning?” in Proceedings of the international conference on learning representations (ICLR), 2026.

[7]

R. A. Rensink, J. K. O’Regan, and J. J. Clark, “To see or not to see: The need for attention to perceive changes in scenes,” Psychological Science, vol. 8, no. 5, pp. 368–373, 1997.

[8]

A. Hollingworth and J. M. Henderson, “Semantic informativeness mediates the detection of changes in natural scenes,” Visual Cognition, vol. 7, no. 1–3, pp. 213–235, 2000.

[9]

G. G. Cole and S. P. Liversedge, “Change blindness and the primacy of object appearance,” Psychonomic Bulletin & Review, vol. 13, no. 4, pp. 588–593, 2006.

[10]

M. J. Wright, “Saliency predicts change detection in pictures of natural scenes,” Spatial Vision, vol. 18, no. 4, pp. 413–430, 2005.

[11]

J. A. Stirk and G. Underwood, “Low-level visual saliency does not predict change detection in natural scenes,” Journal of Vision, vol. 7, no. 10, pp. 1–10, 2007.

[12]

OpenAI, OpenAI technical report“GPT-5 system card.” 2025.

[13]

G. Comanici, E. Bieber, M. Schaekermann, et al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” arXiv preprint arXiv:2507.06261, 2025.

[14]

S. Bai, Y. Cai, R. Chen, et al., “Qwen3-VL technical report,” arXiv preprint arXiv:2511.21631, 2025.

[15]

Kimi Team, T. Bai, Y. Bai, Y. Bao, et al., “Kimi K2.5: Visual agentic intelligence,” arXiv preprint arXiv:2602.02276, 2026.

[16]

GLM-V Team, W. Hong, et al., “GLM-4.5V and GLM-4.1V-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning,” arXiv preprint arXiv:2507.01006, 2025.

[17]

W. Wang, Z. Gao, L. Gu, et al., “InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency,” arXiv preprint arXiv:2508.18265, 2025.

[18]

Microsoft, “Playwright: Fast and reliable end-to-end testing for modern web apps.” https://playwright.dev, 2020.

[19]

A. Radford, J. W. Kim, C. Hallacy, et al., arXiv:2103.00020“Learning transferable visual models from natural language supervision,” in Proceedings of the 38th international conference on machine learning (ICML), 2021.

[20]

OpenAI, S. Agarwal, et al., “Gpt-oss-120b & gpt-oss-20b model card,” arXiv preprint arXiv:2508.10925, 2025.

[21]

Tailwind Labs, “Tailwind CSS: A utility-first CSS framework.” https://tailwindcss.com, 2017.

[22]

Google DeepMind, “Gemini 3.1 Pro model card.” https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-1-Pro-Model-Card.pdf, 2026.

[23]

Google DeepMind, “Gemini 3 Flash model card.” https://deepmind.google/models/model-cards/gemini-3-flash/, 2025.

[24]

OpenAI, “GPT-5.4 thinking system card.” https://openai.com/index/gpt-5-4-thinking-system-card/, 2026.

[25]

Qwen Team, “Qwen3.5-397B-A17B model card.” https://huggingface.co/Qwen/Qwen3.5-397B-A17B, 2026.

[26]

X. Fu, Y. Hu, B. Li, et al., “BLINK: Multimodal large language models can see but not perceive,” in ECCV, 2024, doi: 10.1007/978-3-031-73337-6_9.

[27]

B. Li, Z. Lin, W. Peng, et al., “NaturalBench: Evaluating vision-language models on natural adversarial samples,” in NeurIPS, 2024.

[28]

L. Chen, J. Li, X. Dong, et al., “Are we on the right way for evaluating large vision-language models?” in NeurIPS, 2024.

[29]

S. Tong, E. Brown, P. Wu, et al., “Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs,” in NeurIPS, 2024.

[30]

A. Kamath, J. Hessel, and K.-W. Chang, “What’s ‘up’ with vision-language models? Investigating their struggle with spatial reasoning,” in EMNLP, 2023, doi: 10.18653/V1/2023.EMNLP-MAIN.568.

[31]

J. Cai, K. Yang, L. Fu, et al., “CompareBench: A benchmark for visual comparison reasoning in vision-language models,” arXiv preprint arXiv:2509.22737, 2025.

[32]

J. Zhang, D. Yao, R. Pi, P. P. Liang, and Y. R. Fung, “VLM\(^2\)-bench: A closer look at how well VLMs implicitly link explicit matching visual cues,” arXiv preprint arXiv:2502.12084, 2025.

[33]

M. Ukai, S. Kurita, and N. Inoue, arXiv:2510.22571“STATUS bench: A rigorous benchmark for evaluating object state understanding in vision-language models,” arXiv preprint arXiv:2510.22571, 2025.

[34]

D. Marsili, A. Mehta, R. Y. Lin, and G. Gkioxari, “Same or not? Enhancing visual perception in vision-language models,” arXiv preprint arXiv:2512.23592, 2025.

[35]

R. Awal, S. Ahmadi, L. Zhang, and A. Agrawal, arXiv:2407.16772“VisMin: Visual minimal-change understanding,” in Advances in neural information processing systems (NeurIPS), 2024.

[36]

Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,” in Findings of the association for computational linguistics: EMNLP 2023, 2023, [Online]. Available: https://arxiv.org/abs/2305.10355.

[37]

T. Guan et al., “HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2024, [Online]. Available: https://arxiv.org/abs/2310.14566.

[38]

Z. Sun et al., “Aligning large multimodal models with factually augmented RLHF,” arXiv preprint arXiv:2309.14525, 2023, [Online]. Available: https://arxiv.org/abs/2309.14525.

[39]

J. Wang et al., “AMBER: An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation,” arXiv preprint arXiv:2311.07397, 2023, [Online]. Available: https://arxiv.org/abs/2311.07397.

[40]

F. Zhu, Z. Liu, X. Y. Ng, et al., “MMDocBench: Benchmarking large vision-language models for fine-grained visual document understanding,” arXiv preprint arXiv:2410.21311, 2024, [Online]. Available: https://arxiv.org/abs/2410.21311.

[41]

H.-T. Yu, X.-S. Wei, Y. Peng, and S. Belongie, “Benchmarking large vision-language models on fine-grained image tasks: A comprehensive evaluation,” arXiv preprint arXiv:2504.14988, 2025, [Online]. Available: https://arxiv.org/abs/2504.14988.

[42]

W. Peng, S. Xie, Z. You, S. Lan, and Z. Wu, “Synthesize, diagnose, and optimize: Towards fine-grained vision-language understanding,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2024.

[43]

H. Jhamtani and T. Berg-Kirkpatrick, “Learning to describe differences between pairs of similar images,” in Proceedings of the 2018 conference on empirical methods in natural language processing (EMNLP), 2018, doi: 10.18653/v1/D18-1436.

[44]

Z. Wang, M. Wang, S. Xu, Y. Li, and B. Zhang, “CCExpert: Advancing MLLM capability in remote sensing change captioning with difference-aware integration and a foundational dataset,” arXiv preprint arXiv:2411.11360, 2024.

[45]

Y. Li, W. Xu, Y. Zhang, Z. Wei, and M. Peng, “BTCChat: Advancing remote sensing bi-temporal change captioning with multimodal large language model,” arXiv preprint arXiv:2509.05895, 2025.

[46]

D. H. Park, T. Darrell, and A. Rohrbach, “Robust change captioning,” in Proceedings of the IEEE/CVF international conference on computer vision (ICCV), 2019.

[47]

M. Forbes, C. Kaeser-Chen, P. Sharma, and S. Belongie, “Neural naturalist: Generating fine-grained image comparisons,” in Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), 2019.

[48]

A. Black, J. Shi, Y. Fan, T. Bui, and J. Collomosse, “VIXEN: Visual text comparison network for image difference captioning,” arXiv preprint arXiv:2402.19119, 2024.

[49]

T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” in CVPR, 2023, doi: 10.1109/CVPR52729.2023.01764.

[50]

Q. Jiao, D. Chen, Y. Huang, B. Ding, Y. Li, and Y. Shen, “Img-Diff: Contrastive data synthesis for multimodal large language models,” arXiv preprint arXiv:2408.04594, 2024.

[51]

L. Dunlap et al., “Describing differences in image sets with natural language,” arXiv preprint arXiv:2312.02974, 2023.

[52]

E. Hu, L. Guo, T. Yue, Z. Zhao, S. Xue, and J. Liu, “OneDiff: A generalist model for image difference captioning,” arXiv preprint arXiv:2407.05645, 2024.

[53]

X. Hu et al., “Expert knowledge-aware image difference graph representation learning for difference-aware medical visual question answering,” in KDD, 2023, doi: 10.1145/3580305.3599819.

[54]

K. Zhang, L. Mo, W. Chen, H. Sun, and Y. Su, “MagicBrush: A manually annotated dataset for instruction-guided image editing,” in Advances in neural information processing systems (NeurIPS), 2023.

[55]

M. Hui et al., “HQ-Edit: A high-quality dataset for instruction-based image editing,” arXiv preprint arXiv:2404.09990, 2024.

[56]

S. Sheynin et al., “Emu edit: Precise image editing via recognition and generation tasks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2024, pp. 8871–8879.

Correspondence to zhanglinhao90@gmail.com.↩︎

DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?