1 Introduction↩︎

Visual regression testing (VRT) is a standard quality assurance practice in modern software engineering (SE) [1], [2]. Before each release, user interface (UI) screenshots are captured and compared against an approved baseline to detect unintended visual changes introduced by code, style, or content updates, and every flagged difference is routed to a human reviewer who decides whether it is an intended update or a regression [3]. As release cycles shorten and UIs grow more complex [4], VRT has become a critical safeguard against user-facing regressions. Yet, pixel-by-pixel comparison, widely used in open-source and continuous-integration pipelines [5], is semantically blind. It flags any deviation, treating sub-pixel rendering jitter, anti-aliasing differences, and dynamic advertisement content identically to genuine structural regressions such as missing components, layout restructuring, or altered navigation [3], [6], [7]. Such indiscriminate flagging produces a high false-positive rate, forcing developers and testers to manually inspect large volumes of flagged screenshots at every release and making VRT a significant and recurring source of quality assurance overhead [3], [8]. Industry tools such as Applitools Eyes [9] and Percy [10] have begun incorporating comparison based on machine learning to reduce false positives, but they are proprietary and lack public evaluation. Hence, separation of meaningful regression from visual noise remains broadly underexplored. Recent research has begun applying vision-language models (VLMs) to visual software engineering tasks, such as detecting cross-browser inconsistencies [11] and repairing visual software issues [12], yet none produces a natural language description of UI changes for VRT.

Image difference captioning (IDC) is precisely the missing capability in VRT pipelines. Given a pair of related images, an IDC model localizes what changed, infers the semantic nature of the change, and verbalizes it as a fluent natural language description, producing output that is both interpretable and selective in ways that pixel-level signals are not. The task has been studied extensively in remote sensing [13]–[18] and natural image domains [19]–[24], where it underpins applications ranging from land-cover monitoring [25] to scene-change description [20]. Yet, despite the breadth of prior work, IDC has never been applied to Web UIs. Unlike natural photographs or satellite imagery, which are captured by cameras and characterized by continuous textures and smooth spatial gradients, Web UI screenshots are programmatically rendered artifacts that are highly structured, text-dense, and composed of axis-aligned components with sharp boundaries and strong layout regularity. Unlike the IDC settings above, where the goal is to describe whatever has changed, the Web UI setting inherits VRT’s defining concern of telling genuine regressions apart from noise, so its changes split into two kinds, meaningful and non-meaningful. Meaningful changes are genuine regressions that should be reported to a reviewer, spanning element-level modifications, textual updates, layout restructuring, and thematic redesigns, with representative examples shown in Figure 1. Non-meaningful changes, such as sub-pixel shifts or minor style adjustments, are the visual noise that VRT pipelines must suppress. Whether the modeling paradigm that makes IDC effective in natural image and remote sensing domains carries over to the Web UI setting is an open question, and one that cannot be answered without a domain-specific dataset and benchmark.

We take a direct step toward answering it by introducing Web UI Image Change Captioning (WUICC), a new task that applies IDC to the VRT setting, and release WUICC-bench, the first dataset and benchmark for the task. Answering the question requires, at minimum, controlled Web UI image pairs that reflect realistic change distributions, natural language annotations that capture the semantics of those changes, and a systematic evaluation protocol that enables fair comparison across methods. None of these exist for the Web UI domain, and WUICC-bench is designed to provide all three. We generate the data with a controlled large language model (LLM)-driven mutation pipeline over HTML pages from WebSight [26], a public corpus of LLM-synthesized Web UI pages. Each sample applies exactly one atomic change drawn from a taxonomy of Web UI changes grounded in VRT literature [3], [6], [7], [27] and industry tooling practice [9], [10], and is paired with an LLM-generated caption that human annotators verify. The LLM applies a semantically coherent, taxonomy-controlled change and emits an aligned caption in the same pass, so every mutation comes with ground-truth supervision. Because the LLM is conditioned only on HTML, with rendering delegated to a headless browser, the pipeline scales cost-effectively.

To assess whether existing IDC methods transfer to the Web UI domain, we re-implement eleven representative methods, spanning CNN-based, Transformer-based, and Mamba-based architectures originally developed for natural images or remote sensing, and evaluate them on WUICC-bench with fixed data splits, tokenization, and evaluation metrics so that the methods differ only in their modeling of change. Our contributions are as follows.

1. We propose WUICC, a new task connecting VRT to IDC, opening a new research direction at the intersection of SE and computer vision and providing a formal problem definition for an important but previously unaddressed SE challenge. We are explicit that the novelty lies in the task framing, the change taxonomy, and the benchmark construction rather than in a new captioning architecture, and we reuse existing IDC methods unchanged so that the reported gap reflects the domain rather than our own modeling choices.

2. We construct and publicly release WUICC-bench through a controlled and verifiable LLM-driven mutation pipeline over Web UI pages. Each sample applies exactly one atomic change drawn from a 37-rule taxonomy, is paired with an automatically generated natural language caption, and passes human verification. We report the pipeline’s success rate and inter-annotator agreement to characterize its reliability. The resulting dataset spans nine meaningful change categories, including element addition and removal, layout restructuring, reordering, and thematic redesigns, together with three non-meaningful categories representing visual noise. Beyond the released data, the pipeline is itself a reusable methodology for building change captioning datasets in structured-UI domains rather than a one-off construction effort.

3. We develop a Web UI change taxonomy grounded in VRT literature and industry tooling practice, and we make explicit the design criteria we propose for it, namely coverage of the changes VRT encounters, separability between categories, and one atomic change per sample.

4. We provide the first empirical benchmark for the WUICC task by re-implementing and evaluating eleven representative IDC methods spanning CNN-based, Transformer-based, and Mamba-based architectures, together with two zero-shot general-purpose VLMs. Our results indicate that both the trained methods and the VLMs tend to struggle in the Web UI domain, generally underperforming relative to IDC scores on natural image and remote sensing benchmarks, yet the trained methods already suppress non-meaningful changes far more selectively than pixel-level comparison, the capability VRT most needs. We offer WUICC-bench as a shared foundation for future domain-specific research.

The remainder of the paper is organized as follows. Section 2 reviews background on VRT and IDC and characterizes the challenges of the WUICC problem. Section 3 describes WUICC-bench. Section 4 presents the experimental setup, results, and ablation studies. Section 5 presents discussion and threats to validity. Section 6 situates our work within the broader literature, and Section 7 concludes the paper.

2 Background↩︎

2.1 Visual Regression Testing↩︎

VRT pipelines re-render UI screenshots after a code change and compare each one against its approved baseline to detect unintended visual regressions, surfacing every flagged difference to a human reviewer for an approve-or-reject decision.

Industry tools have advanced the state of practice considerably [5], with Applitools Eyes [9] performing context-aware comparison beyond pixel matching, Percy [10] suppressing common rendering artifacts in continuous-integration pipelines, and Chromatic [28] detecting regressions at the component granularity. Yet none produces a natural language description of the detected change. WUICC produces exactly such a description, returning a flagged change in words rather than as a raw visual difference for the tester to interpret.

2.2 Image Difference Captioning↩︎

IDC generates a natural language description of the semantic differences between a pair of related images. A model must localize the changed regions, infer the semantic nature of each change, and verbalize it as a fluent sentence. Two domains have driven most prior research, natural image and remote sensing. In the natural image domain, the inputs are photographs of the same scene under different conditions. In the remote sensing domain, known as Remote Sensing Image Change Captioning (RSICC), the inputs are satellite or aerial images captured at two different time points.

Both settings share a two-stage formulation in which a visual stage extracts change-aware features from the image pair and a language stage decodes these features into a caption [29], [30]. Web UI screenshots, however, differ substantially from both in visual statistics, layout regularity, text density, and the notion of a meaningful “change”, raising the question of whether these methods transfer to the Web UI domain, which our benchmark is designed to answer.

3pt 1pt 1pt

2.3 Challenges in WUICC↩︎

The Web UI setting introduces difficulties absent from prior IDC domains. We identify four core challenges that shape the difficulty of WUICC.

1. Layout diversity. Web UI layouts follow no single visual pattern, varying greatly across pages in structure, element arrangement, and visual style. Combined with the wide variety of change types covered by the taxonomy, a model must handle an extremely large space of layout-change combinations with no fixed template to rely on, making generalization substantially harder than in domains where the visual background is more consistent.

2. Verbatim text reproduction. A substantial fraction of UI changes are textual, affecting labels, headings, or body text. Unlike object-level changes in natural scenes, where paraphrasing is acceptable [20], text in a UI must be transcribed verbatim. For example, replacing “Submit” with “Send” is a meaningful change, and the caption must reproduce the exact text “Send”. Any other word, however semantically similar, constitutes an incorrect description, pushing the task beyond conventional captioning toward reading the rendered text and reproducing the changed content without any error.

3. High-resolution images with fine-grained changes. Web UI screenshots are typically high-resolution, and a meaningful change often spans only a small fraction of the canvas. Detecting and describing changes at such granularity demands finer spatial sensitivity than the coarser changes typical of prior natural image IDC [20] and RSICC [30], [31] domains, where the altered content often occupies a large salient region of the scene.

4. Non-meaningful change suppression. The WUICC setting includes non-meaningful modifications rooted in web rendering characteristics, such as sub-pixel shifts, minor spacing adjustments, and subtle style variations, which are defined by VRT practice as visual noise that should not be reported. Unlike noise in natural image or remote sensing settings, which arises from environmental factors such as lighting or seasonal variation [32], web rendering noise stems from deterministic and well-documented rendering mechanisms such as anti-aliasing, sub-pixel positioning, and font rasterization. Industry VRT tools attempt to suppress such noise [9], [10], [28], but how effectively they do so is not publicly evaluated. A change captioning model, by contrast, must learn to suppress these artifacts and report “no change” regardless of whether the modification is visually detectable.

3 The WUICC-bench↩︎

WUICC-bench is constructed through a controlled LLM-driven mutation pipeline, illustrated in Figure 2. The pipeline is organized around a change taxonomy that defines what counts as a meaningful or non-meaningful UI modification and that governs both what changes are generated and how they are described. Because the taxonomy is the central design artifact, we present its rationale and categories first, then describe the generation procedure, human verification, and the source corpus from which the data is drawn.

3.1 Design Rationale for the Change Taxonomy↩︎

Criteria. A good taxonomy should meet certain design criteria, but no universally agreed set of them exists, so we identify the three we consider essential for WUICC. First, coverage requires that the meaningful categories span the regression types that VRT tools and human testers are designed to surface, so the dataset does not systematically omit practically relevant changes. Second, separability requires that the boundary between meaningful and non-meaningful changes and between different meaningful rules be operationalizable by the LLM at generation and by humans at audit, otherwise samples become ambiguous and verification collapses. Third, single-label composition requires that each sample contain exactly one change. It keeps evaluation clean, since a model’s success or failure on a sample can be traced to one specific change rather than to a mix of changes, and it keeps human verification tractable, since an annotator only needs to check that the caption completely and accurately describes a single change rather than disentangle several simultaneous edits.

Sources informing the taxonomy. To make the taxonomy as exhaustive as possible rather than an arbitrary set, we draw candidate Web UI change types both from repeatedly prompting an LLM to propose further changes until it surfaces no substantially different ones and from the changes documented in established VRT literature [3], [6], [7], [27] and in practitioner accounts of industry VRT practice [5]. We then consolidate the combined set into mutually exclusive categories that reflect the changes VRT actually encounters. Design choices. To keep the categories clearly separable and aligned with how regressions appear in practice, we also made the following design choices. First, we split element removal from addition rather than collapsing them into a single category. Second, we treat thematic changes as meaningful but pure style changes as non-meaningful. The difference is global-intentional, such as a light-to-dark mode switch, versus local-cosmetic, such as a single shadow tweak, mirroring how industry tools separate theme regressions from cosmetic style noise. Third, although layout changes and resizing are both meaningful and both spatial, we keep them as separate categories because their visual signatures differ, with layout changes affecting inter-element relationships and resizing affecting intra-element dimensions.

Encoding. We turn each criterion into a concrete prompt constraint. To serve coverage, the taxonomy is realized as a set of numbered atomic rules, so generation spans the full range of changes a Web UI may undergo. Separability is served at two boundaries. Between the meaningful and non-meaningful sides, every non-meaningful sample reuses a description drawn verbatim from a fixed phrase list, which stops the LLM from inventing spurious differences and keeps the two sides apart. Between individual rules, the atomic and mutually exclusive rule definitions let an auditor assign each sample to exactly one rule, and because meaningful changes are described by what is visible on the page rather than by markup-level selectors, that audit can be done on the rendered screenshot without inspecting the HTML. To serve single-label composition, the LLM is instructed to apply exactly one specified rule per sample, making the taxonomy single-label by construction. Compound regressions are out of scope of our work and recorded as a limitation in Section 5.1.

Table 1 summarizes the twelve categories encoded in the prompt. Nine are meaningful categories, whose changes must be described in the generated caption, and three are non-meaningful categories, whose samples carry a fixed “no change” description.

3.2 LLM-Driven Sample Generation↩︎

Figure 2 summarizes the pipeline that turns a source page into a verified sample through four stages.

(1) Source HTML. Each sample starts from a source HTML page and the change taxonomy of 37 atomic rules. For every sample we draw one rule uniformly from the taxonomy and feed it to the mutator alongside the page.

(2) LLM Mutator. Prompted with the page HTML and the selected rule, the LLM applies that single change and, in one pass, emits both the mutated HTML and a natural language description of the change. Producing the edit and its description together keeps them aligned by construction, avoiding the drift a separate captioning pass would introduce. For a non-meaningful rule the description reuses a phrase drawn from a small fixed list of interchangeable no-change phrases, which stops the model from inventing spurious differences.

(3) Headless Renderer. The original and the mutated HTML are rendered under an identical viewport, producing images \(I_{\text{pre}}\) and \(I_{\text{post}}\), which differ only in the intended change.

(4) Annotator Review. Annotators review the candidate pair against its description and either keep the sample for WUICC-bench or discard it. Section 3.3 details the verification stage and the failure modes it targets.

Because the pipeline is driven entirely by the prompt and indexed by atomic rules, it scales to large corpora without manual authoring of changes. Both the taxonomy and the rule-indexed pipeline are thus reusable beyond WUICC-bench, supporting extension to new change types and the construction of similar benchmarks in other structured-UI domains.

3.3 Human Verification↩︎

Generated samples inevitably contain failure cases, and a human verification stage is therefore introduced to filter them out. Through manual inspection of a large pool of generated samples, we identify three recurring failure modes. In the first, the LLM fails to apply the intended edit to the HTML, so the rendered image pair shows no change while the description still claims one. In the second and most common, the LLM applies the edit and its textual description is consistent with it, but the rendered image pair still fails to reflect the intended change. In the third, the textual description does not faithfully describe the HTML edit, so although the image pair is correctly rendered from the edit, the description does not state the change it actually shows. To address both, each sample is reviewed by one of two annotators, given only the image pair and the description, both of whom were trained on the change taxonomy and the review criterion before annotation. A sample is retained only if its caption completely and accurately describes the change visible in the rendered pair, and discarded otherwise.

3.4 LLM Selection↩︎

We use GPT-5.1 [33] as the mutator for its established strong performance on challenging reasoning and code-related tasks, which our pipeline relies on to apply a structural HTML edit and describe the resulting change accurately. Among the available options, the larger GPT-5.5 is more capable but substantially more expensive, whereas the smaller GPT-5-mini is cheaper but a weaker model, so GPT-5.1 offers a reasonable balance of capability and cost. Because the pipeline conditions only on HTML and never incurs vision tokens, the resulting cost is low, making large-scale construction economical.

3.5 Dataset Statistics and Comparison↩︎

WUICC-bench is built from the 37 atomic rules of our change taxonomy. Each of these rules is sampled with equal probability during generation, giving an intended uniform distribution over rules. For each atomic rule, only a small fraction of generated samples are invalid, so the distribution is largely preserved. The final dataset retains 9,906 valid samples, of which 8,583 contain meaningful changes and 1,323 contain non-meaningful changes.

Table ¿tbl:tab:resolution? summarizes the resolution distribution of the screenshots, where each sample contributes a before and after screenshot and each row reports a resolution group (width \(\times\) height) with the height binned into intervals. All pages are rendered at a fixed viewport width of 1280 pixels, and about two thirds of the screenshots match the standard \(1280\times720\) viewport. The remainder are full-page captures whose height grows with page content, extending to 4320 pixels, with only 0.3% of screenshots wider than 1280 pixels. The screenshots are thus consistently high-resolution, so a single change occupies only a small fraction of the canvas, realizing the challenge of fine-grained changes identified in Section 2.

Figure 3 shows the distribution of valid word counts per caption. The sharp peak at four words corresponds to the fixed short phrases attached to non-meaningful samples, whereas the captions of meaningful changes form a broad distribution that peaks around 18 words and decays slowly, with 102 captions exceeding 50 words. Models must therefore handle both terse “no change” outputs and long, detailed change descriptions.

We then split the dataset into training, validation, and test sets with a ratio of approximately 70:10:20, giving 6,963 training, 982 validation, and 1,961 test samples. Two properties of the resulting dataset are worth highlighting, described below.

a) Caption style. Because descriptions are produced by an LLM rather than written by a specific or handful human annotators, they exhibit substantial lexical and syntactic variability, reaching up to 113 valid words. They are also considerably longer than typical captions in the natural image or remote sensing settings. The same type of change can be described in multiple valid phrasings, as shown in Fig. 4, more faithfully reflecting how human testers would phrase the same regression in different reports.

b) Scale. WUICC-bench contains 9,906 samples, substantially larger than change captioning datasets such as Dubai-CC [31] (500 pairs) and Image Editing Request [34] (3,939 pairs), and comparable in size to larger human-annotated benchmarks such as Spot-the-Diff [20] (13,192 pairs) and LEVIR-CC [30] (10,077 pairs), a scale sufficient to support benchmarking.

c) Distinguishing properties. Beyond scale, WUICC-bench differs from prior datasets on two axes that matter for VRT, namely its programmatically rendered, pixel-aligned, viewpoint-free Web UI domain, and its verbatim-text requirement under which a textual edit must be transcribed exactly rather than paraphrased.

3.6 Source HTML Corpus↩︎

Having characterized the resulting dataset, we now turn to the source corpus its pages are drawn from and the rationale behind it. The source HTML is drawn from WebSight [26], which spans a wide range of Web UI structures and styles. WebSight consists of LLM-synthesized pages designed to resemble real Web UIs. In its released form, the corpus is self-contained static HTML that renders deterministically, which is essential for a benchmark in which each pre/post pair must differ only in the intended modification.

We build on the released corpus rather than scraping production sites ourselves so that the resulting benchmark can be openly redistributed. Scraped production HTML is typically not redistributable under copyright and terms-of-service constraints, whereas WebSight is released under terms that permit us to re-render, mutate, and release the derived artifacts. The residual gap to live production traffic is acknowledged in Section 5.1.

From the source corpus, we select pages that are complex along two dimensions, layout and interaction. Layout complexity is measured by the number of top-level UI blocks, and interaction complexity by the number of clickable elements such as buttons. These criteria filter out overly simple pages and retain ones with multi-column or grid-based layouts, nested structures, and multiple functional regions, which offer rich potential for diverse modifications.

3.7 Task Formulation↩︎

Having described how WUICC-bench is constructed, we now formalize the task it is designed to support, which applies IDC to the Web UI domain. Given an ordered pair of Web UI screenshots, a baseline \(I_{\text{pre}}\) and a candidate \(I_{\text{post}}\) rendered from the original and the modified page, the goal is to learn a mapping \(f: (I_{\text{pre}}, I_{\text{post}}) \rightarrow Y\) that generates a natural language caption \(Y=\{y_1,\ldots,y_T\}\) with \(y_t \in \mathcal{V}\) describing the change from \(I_{\text{pre}}\) to \(I_{\text{post}}\), where \(\mathcal{V}\) is the vocabulary and \(T\) the caption length. The formulation turns the output of regression checking from a semantically blind difference signal into a natural language description, so that a tester is told what changed in words instead of being handed a highlighted region to interpret. It also makes the substantial body of IDC methods developed for natural images and remote sensing directly applicable to VRT for the first time, enabling a concrete study of whether their modeling paradigms transfer to the Web UI setting. Models are trained by maximizing the likelihood of the ground-truth caption under a standard token-level cross-entropy loss: \[\mathcal{L} = - \sum_{t=1}^{T} \log p(y_t \mid I_{\text{pre}}, I_{\text{post}}, y_{<t}), \label{eq:ce}\tag{1}\] where \(p(.)\) denotes probability. While some prior works augment the objective with auxiliary losses, such as a cross-view contrastive loss in SCORER [22] or attention and entropy regularization in VARD [21], Eq. (1 ) remains the fundamental training signal common to virtually all change captioning models [13], [15], [29], [30], [35]–[38].

4 Experiments↩︎

4.1 Research Questions↩︎

Our experiments aim to answer the following important research questions (RQs):

• RQ1: Does our LLM-driven generation pipeline produce a reliable benchmark, i.e., do the mutated HTML and its paired change caption yield valid samples that human annotators judge consistently?

  To answer it, we measure the inter-annotator agreement and the accuracy of the LLM-driven generation pipeline, requiring the mutated HTML to render the intended change and the paired caption to describe it correctly.

• RQ2: How well do existing IDC methods, originally developed for remote sensing and natural images, and general-purpose LLMs perform on WUICC?

  To answer it, we evaluate eleven representative IDC methods and two zero-shot general-purpose LLMs on WUICC-bench.

• RQ3: Can the benchmarked methods suppress non-meaningful changes and report “no change”, avoiding the false positives that pixel-level comparison produces in VRT?

  To answer it, we measure the no-change and change accuracy on the non-meaningful and meaningful categories, respectively, and compare against pixel-level baselines.

• RQ4: How do key configuration factors, namely input image resolution and beam size, affect performance?

  To answer it, we run ablation studies on input resolution and beam size with representative methods.

4.2 Benchmark Methods↩︎

We benchmark eleven representative methods spanning RSICC and natural IDC, all re-implemented and trained on WUICC-bench. Each re-implementation follows the architecture and hyperparameters reported in the original work and builds on the authors’ released code where it is available. We release all re-implementations and configurations so that their correctness can be independently verified. In addition to these trained methods, we evaluate two general-purpose VLMs, Llama-3.2-11B-Vision-Instruct [39] and Qwen2-VL-7B-Instruct [40], in a zero-shot setting as reference baselines. We summarize each trained method below.

• Chg2Cap [29]. Siamese CNN backbone with hierarchical self-attention encoder and Transformer decoder.

• SEN [35]. Contrastive-pretrained single-stream visual extractor with a difference encoder guided by cross-attention.

• ICT-Net [13]. Multi-scale CNN features with interactive change-aware encoder and cross gated-attention decoder.

• RSICCformer [30]. CNN with dual-branch Transformer encoder and difference-aware cross-attention.

• RSCaMa [36]. CLIP image encoder with stacked Mamba CaMa layers and Transformer decoder.

• SparseFocus [37]. ResNet-101 backbone with sparse focus attention encoder and Transformer decoder.

• DAE [15]. Dynamic asymmetric encoder via difference-guided dynamic convolution and temporal modeling.

• RMNet [38]. Dual-stream CNN–VMamba network with difference recalibration to suppress pseudo-changes.

• CARD [41]. Decomposition of image-pair features into shared and change-specific components under consistency and independence constraints.

• VARD [21]. Context- and position-aware representations with change isolation based on feature subtraction.

• SCORER [22]. Contrastive view-invariant representations with cross-modal backward reasoning.

Despite their architectural differences, these methods share a common structure. Each pairs a feature extraction backbone, a CNN such as ResNet-101 or a vision Transformer, with a change-aware encoder that models the differences between the image pair and a Transformer-based caption decoder. The change-aware encoder, which sets change captioning apart from single-image captioning, is where the methods differ most.

The two VLM baselines are evaluated zero-shot, with no training on WUICC-bench. Each receives the before and after screenshots together with a fixed instruction to describe the change in one caption, and its output is scored under the same metrics and “no change” convention as the trained methods. They establish how far an off-the-shelf VLM reaches on WUICC without any domain adaptation.

4.3 Evaluation Metrics↩︎

Traditional VRT evaluation is detection-oriented, measuring regression-versus-noise classification or the reduction in differences a tester must confirm [3], [6], [8]. Such metrics score a binary flag or a highlighted region and cannot assess the content of a generated description. We therefore adopt the standard language-level metrics from IDC, which give a reproducible and widely used basis for comparison with prior work. Designing UI-specific metrics that capture verbatim text fidelity is an important direction for future work.

We report the standard language-level captioning metrics, namely BLEU-n (n=1,2,3,4) [42], METEOR [43], ROUGE_L [44], CIDEr [45], and SPICE [46]. They differ in the kind of overlap they reward. BLEU scores n-gram precision and ROUGE_L scores longest-common-subsequence overlap, METEOR adds stem, synonym, and paraphrase matching, CIDEr weights n-grams by TF-IDF so that distinctive content counts more, and SPICE compares scene-graph tuples of objects, attributes, and relations.

4.4 Implementation Details↩︎

Unless stated otherwise, each before/after screenshot is resized to \(512\times512\). The baselines were originally designed for \(256\times256\) inputs, which is too coarse to resolve the dense text and fine-grained changes in WUICC-bench. We therefore raise the resolution, but adopt \(512\times512\) rather than a higher resolution, since it captures the detail that \(256\times256\) misses while remaining computationally tractable for every baseline, unlike \(768\times768\), which our ablation (Table 7) finds most accurate but contains \(2.25\times\) as many pixels, raising compute and memory cost accordingly. We further examine the effect of input resolution in the ablation study. The captions are truncated to at most 50 valid words, which affects only the 102 captions (about 1% of the dataset) that are longer. Each baseline keeps its original architecture and published hyperparameters, including optimizer, learning rate, and batch size, while sharing the same data pipeline, tokenization, and metric implementation. Each method is optimized with its original training objective from the corresponding paper, namely the cross-entropy loss of Eq. 1 together with any method-specific auxiliary losses, on a single NVIDIA H100 GPU. At inference, captions are generated by beam search with a width of 3, and all reported numbers are computed on the test split.

4.5 Quantitative Results↩︎

4.5.1 Agreement Rate↩︎

To assess benchmark reliability, we first present the results of human verification. We select 100 samples for evaluation, deliberately over-sampling invalid generations so that valid and invalid cases are both well represented, which avoids the inflated chance agreement that a heavily skewed sample would produce. Each sample is independently annotated by two annotators, and inter-annotator reliability is measured with Cohen’s Kappa. The two annotators judge 48 and 60 of the 100 samples as valid, respectively, and agree on 86 samples, with 47 judged valid and 39 judged invalid by both. As reported in Table 2, the result is \(\kappa \approx 0.722\), which corresponds to substantial agreement [47] and suggests that the annotations are reliable.

4.5.2 LLM Accuracy↩︎

We then present the accuracy of the LLM-driven generation, computed over the entire pool of generated samples rather than the 100-sample agreement subset, and defined as the proportion that the verification stage of Section 3.3 retains as correct. The failure cases fall into three types. First, the LLM fails to modify the code, which is attributed to the limitations of the LLM itself. Second, the LLM successfully modifies the code, but the rendered UI image does not reflect the intended change, which is attributed to the UI rendering process. Third, the LLM successfully modifies the code, but the generated caption fails to accurately describe the change, which is attributed to the captioning process. As reported in Table 3, the pipeline retains 78.09% of generated samples as correct, which indicates its effectiveness in automating the task and reducing the manual annotation cost. Notably, the 78.09% and the agreement coefficient \(\kappa\) capture different aspects of reliability. The acceptance rate measures how often the pipeline yields a correct sample over the full pool, while \(\kappa\) measures how consistently annotators apply the verification criterion on the 100-sample subset. The two are therefore complementary rather than measurements of the same quantity.

4.5.3 Results on WUICC-bench↩︎

With the benchmark validated, we now examine method performance by comparing the eleven trained IDC methods on WUICC-bench, together with the two zero-shot VLM baselines, Llama-3.2-11B-Vision-Instruct and Qwen2-VL-7B-Instruct. Table 4 reports the comparison, and among the trained methods no single one dominates. SparseFocus [37] attains the best n-gram and CIDEr scores, reaching 0.2571 BLEU-4 and 2.0761 CIDEr, with Chg2Cap [29] close behind. On SPICE the ordering flips, as Chg2Cap is best at 0.3345 while SparseFocus falls to near the bottom at 0.2579. The divergence is itself informative, because it shows that strong lexical overlap on n-gram measures does not guarantee semantically adequate captions, given that SPICE scores agreement over scene-graph propositions of objects, attributes, and relations rather than over surface word sequences. A second and orthogonal pattern is architectural rather than metric-specific, and it tracks the domain assumptions embedded in each method’s change-aware encoder. The three methods originally engineered for natural-image IDC, namely the viewpoint-robust and disentanglement designs of SCORER [22], VARD [21], and CARD [41], all settle into the lower half of the ranking across both n-gram and SPICE metrics. Because Web UI screenshots are programmatically rendered and therefore pixel-aligned, free of the camera viewpoint shifts that these designs are built to neutralize, the additional invariance machinery they carry adds parameters and optimization burden without addressing any difficulty that the Web UI setting actually presents. The two zero-shot VLMs trail the trained methods on most metrics, with Llama-3.2-11B scoring only 0.0363 BLEU-4 and Qwen2-VL 0.0920, although Qwen2-VL reaches a competitive 1.8699 CIDEr, showing that a general-purpose VLM can capture distinctive changed content yet still falls short of the n-gram and SPICE fidelity the trained methods achieve. Overall the absolute scores remain low across both groups, indicating the difficulty of our benchmark for the trained-from-scratch IDC paradigm and for off-the-shelf VLMs alike.

4.5.4 Change Awareness↩︎

A core requirement for VRT is false positive suppression, namely suppressing visual noise rather than reporting it, without becoming blind to genuine changes. We therefore report two accuracies in Table 5. “No-Change Acc.” is the suppression accuracy on non-meaningful samples, the proportion correctly described as “no change”. “Change Acc.” is the proportion of meaningful samples correctly reported as changed rather than suppressed. Higher is better for both, and reporting them together guards against a degenerate model that trivially emits “no change” everywhere. The goal here is not to rank all methods on suppression, which Table 4 already covers, but to contrast the captioning paradigm with pixel-level comparison, so we report four IDC methods that span the architectural families in our benchmark, the CNN-based SEN and ICT-Net, the dual-branch Transformer RSICCformer, and the CNN-VMamba RMNet, together with the two zero-shot VLMs rather than all eleven IDC methods. As a reference we include a naive pixel-level comparison (“Pixel Diff”) and one that applies a 0.1 difference threshold with anti-aliasing tolerance (“Pixel Diff (w/ tol.)”).

The two pixel-diff references illustrate a known weakness of pixel-level comparison. Both flag almost every meaningful change, reaching a change accuracy of 100.00% and 97.93%, yet they suppress almost nothing, with a no-change accuracy of only 0.00% and 6.72%. Any sub-pixel shift, minor restyle, or dynamic-content update is therefore reported as a difference, the kind of false positive that VRT pipelines must triage by hand. Adding an anti-aliasing tolerance raises suppression by under seven points and already begins to miss genuine changes, suggesting that a single fixed threshold struggles to separate noise from signal.

The four trained IDC methods behave in the opposite and desirable way. They suppress the large majority of non-meaningful changes, with a no-change accuracy between 85.82% and 97.39%, while still reporting most genuine changes, with a change accuracy between 93.55% and 96.16%. RMNet and ICT-Net reach the highest suppression among the IDC methods at 97.39% and 97.01%, and none collapses to the degenerate solution of always emitting “no change”, since every IDC change accuracy stays above 93%, suggesting that these models capture change at a more semantic than pixel level. The two zero-shot VLMs behave far less stably. Qwen2-VL suppresses almost all non-meaningful changes, reaching 99.63% no-change accuracy, but does so by under-reporting genuine ones, with its change accuracy falling to 65.68%, whereas Llama-3.2-11B fails in the opposite direction with only 21.64% no-change accuracy. Their change accuracies, 65.68% and 79.39%, both fall well below those of the trained methods, indicating that off-the-shelf VLMs do not yet provide the selective suppression VRT needs.

4.6 Ablation Studies↩︎

We study the sensitivity of performance to two design factors, input image resolution and beam size.

4.6.1 Influence of Input Image Resolution↩︎

Since our screenshots are high-resolution, the input resolution fed to the models may affect their performance. We therefore study its impact by evaluating three settings, \(256\times256\), \(512\times512\), and \(768\times768\), which probe the sensitivity of each model to visual granularity in Web UI change understanding. We run the ablation on five representative methods, with results reported in Table 7. Across all methods, the best scores consistently occur at \(768\times768\), and \(512\times512\) is usually second best, indicating that higher input resolution generally yields better performance, but higher resolution also requires more compute and memory, which is why we adopt \(512\times512\) as the default rather than \(768\times768\).

4.6.2 Effect of Beam Size↩︎

We further investigate how beam search size during inference affects caption quality. We consider three settings, i.e., 1, 3, and 5, and report results on three representative methods at a fixed resolution of \(512 \times 512\). As shown in Table 6, larger beam sizes do not uniformly help. Across methods, beam size 1 generally yields the best BLEU-n and METEOR scores, whereas beam size 3 is usually best on ROUGE_L, CIDEr, and SPICE, and remains consistently strong overall, which is why we adopt it as the default. Beam size 5 brings no further gain, suggesting that moderate beam search suffices for WUICC.

5 Discussion↩︎

Implications for VRT practice. Change captioning supplies the capability VRT pipelines lack. Where a pixel diff returns an undifferentiated mask and floods the pipeline with false positives, a caption directly localizes and names the changed element while suppressing non-meaningful changes more selectively, augmenting VRT triage with a natural language description of what changed.

Implications for future modeling. Web UI changes are small relative to the canvas and frequently textual, so progress depends on capabilities the natural image and remote sensing benchmarks do not exercise, namely verbatim text reading and fine-grained localization of a small changed region within a large unchanged canvas [48]. Building the WUICC-specific architecture these axes call for is left to future work that our benchmark is designed to enable.

5.1 Threats to Validity↩︎

Construct validity. Our rankings rely on captioning metrics (BLEU, METEOR, ROUGE_L, CIDEr) that reward surface lexical similarity. Verbatim reproduction of text edits is partially captured, since CIDEr and high-order BLEU reward exact, TF-IDF-weighted matching of the changed text and penalize substituted tokens, whereas METEOR and SPICE credit synonyms and so do not reflect verbatim correctness. Even so, because the changed text is a small fraction of a caption, a single-token error is diluted. These metrics nonetheless remain the standard, reproducible basis for comparison in IDC, and our conclusions do not rest on any single one.

Internal validity. Because the ground-truth captions are produced by the LLM-driven pipeline rather than written by humans, models scored under lexical metrics could be favored for sharing the generator’s phrasing. We mitigate the circularity in three ways. First, trained annotators verify every sample, checking that the caption completely and accurately describes the change in the rendered pair, including verbatim correctness of any changed text, with substantial agreement (\(\kappa \approx 0.722\)) and 78.09% of generations accepted, so fitting these references means learning accurate change descriptions. Second, the bias cannot affect the rankings our conclusions rest on, since all eleven benchmarked methods are trained from scratch on the same references and the two zero-shot baselines (Llama and Qwen) come from a different family than the GPT-family generator, which leaves only the absolute scores at risk. Third, our change and no-change accuracies reward correct content rather than wording, and corroborate the metric-based trends.

External validity. Our source pages come from WebSight [26], a corpus of LLM-synthesized HTML curated for general UI training rather than the framework-heavy production sites where visual regressions matter most, so the layouts, components, and dynamic JavaScript-driven content of WUICC-bench need not match real production UIs. Each sample also contains exactly one taxonomy-drawn change, a deliberate choice for clean attribution and tractable verification rather than an oversight (Section 3.1), whereas real regressions sometimes involve multiple interacting changes within a single release, an extension we leave to future work. Finally, our coverage of LLM-based change captioning is partial. Specialized systems such as Change-Agent [49], ChangeChat [50], and TEOChat [51] require supervision from change masks or bounding boxes that WUICC-bench does not provide, so we do not evaluate them. We do include two open general-purpose VLMs, Llama-3.2-11B-Vision-Instruct and Qwen2-VL-7B-Instruct, as zero-shot baselines, and find that they fall short of the trained methods on both caption quality and selective suppression. A broader sweep of stronger and closed-source VLMs such as GPT-5 and Claude is costly to query with two high-resolution images, and their zero-shot quality varies widely across models, so a systematic study of these larger VLMs is left to future work. Quantifying the production gap with a held-out probe set and extending the pipeline to compound changes are also next steps.

6 Related Work↩︎

Visual Regression Testing. Academic VRT research reduces the manual confirmation burden of pixel-level comparison through grouping identical differences [3], region-based matching [6], masking dynamic content areas [8], and hybrid image-HTML comparison [7], with a complementary line on dependent tests that reduces flaky failures [52]. These output binary flags or highlighted regions, whereas our work produces a natural language description that such detection-oriented techniques cannot. Recent VLM work detects cross-browser inconsistencies [11] and repairs visual issues [12], but targets detection and repair rather than describing the change.

Remote Sensing Image Change Captioning. RSICC research has progressed through several methodological waves: early pipelines that couple a convolutional visual encoder with a recurrent or support vector machine based caption generator [31], [53]–[58], diffusion-based generation [59]–[62], Transformer- and attention-based architectures [13], [29], [30], [35], [37], [63]–[67], Mamba state-space models for linear-complexity sequence modeling [36], [68], [69], and most recently LLM-based approaches [49]–[51], [70]–[73]. Our benchmark adopts representative methods from each wave as baselines.

Image Difference Captioning. IDC on natural images has explored change-region identification [74], contrastive pre-training [19], cross-view representation reconstruction [22], viewpoint-robust disentanglement [21], common-difference feature decoupling [41], regional difference modeling [48], multi-scale perception in multimodal LLMs [24], prompt-to-prompt synthetic pairs [23], generalist VLMs with visual-delta modules [75], BLIP2 adaptation [76], and describing differences across image sets [77]. Likewise, we select representative methods for our benchmark.

7 Conclusion↩︎

WUICC-bench is the first dataset and benchmark for WUICC, the task of describing Web UI changes in natural language for VRT, built by a controlled LLM-driven mutation pipeline with human-verified captions. Evaluating eleven representative methods, we find that existing models transfer poorly to the Web UI domain, yet already suppress non-meaningful changes more selectively than pixel-level comparison, the capability VRT most needs. Our results position change captioning as a promising paradigm for semantic, more informative VRT and a foundation for future domain-specific methods.

References↩︎