1 Introduction↩︎

User interface design shapes how users perceive, navigate, and engage with digital interfaces. Small changes in layout, wording, visual hierarchy, or icons can lead to measurable differences in experience, preference, and behavior [1]–[3]. Evaluating these effects usually requires user studies or A/B experiments, where participants answer questions about a product or choose between design variants. Collecting such evidence is valuable but costly, resulting in many interfaces being released with limited formal evaluation [4], [5].

Recent work has explored using Multimodal Large Language Models (MLLMs) as automatic evaluators. One line of work predicts interface quality from screenshots, either by training models on synthetically degraded UI pairs [2] or by prompting MLLMs with design heuristics [6]–[8]. Another line studies whether vanilla MLLMs can predict average human preferences between two UI variants [3], [4]. While these methods provide useful signals, they usually evaluate interfaces from a model-centric perspective. Few-shot prompting can reflect the MLLM’s own preferences or biases, while supervised fine-tuning on answers does not explicitly model why a particular user would select one answer over another, limiting agent’s ability to generalize to new interfaces, questions, and user profiles. This drawback is critical because UI judgments are often user-dependent. The same interface may be perceived differently depending on a user’s goals, prior experience, visual preferences, or domain familiarity. A useful surrogate evaluator should therefore go beyond predicting an average preference and simulate how a specific user would respond to a given UI/UX evaluation question.

We introduce PerceptUI, a framework for persona-conditioned UI/UX evaluation. Given a UI screenshot, a user persona, and a question, PerceptUI predicts how the corresponding user would answer and generates a natural-language rationale for the prediction. Our approach is motivated by two observations. First, imitation learning is a weak training signal as it tells the model what to output but not why the correct option fits, and the other options do not. We therefore leverage a teacher model to turn each example into a contrastive reflection that justifies both the chosen answer and the rejection of each alternative, and use these traces to fine-tune a vision-language model. Second, evaluation questions vary substantially across surveys in wording, answer scales, and required interpretation. Thus, we apply a reflective prompt-evolution step, where errors made by the model are summarized and used to refine the evaluation prompt. This step adapts the inference prompt to recurring failure patterns without relying on manual survey-specific prompt tuning. Across multiple UI/UX evaluation tasks, PerceptUI reaches human-level performance, generalizes to unseen questions and personas, and produces calibrated population-level answer distributions. Overall, the framework supports both early-stage evaluation and targeted analysis of how different designs affect different users.

2 Related Work↩︎

UI Evaluation with MLLMs. Automatic UI evaluation has long studied computational measures of visual complexity, aesthetic structure, and layout preference from interface properties [9], [10]. More recent work uses vision-language models to evaluate UI screenshots directly. One line of work trains dedicated models for design-quality prediction. For example, UIClip [2] fine-tunes CLIP on BetterApp, a dataset of designer-rated mobile UI comparisons. Another line of work prompts MLLMs to produce natural-language critiques from design heuristics or visual evidence. [6] query GPT-4 with Nielsen-style heuristics, UICrit [7] studies critique generation from expert mobile-UI feedback, and [8] improve grounding through region-level visual prompts. Beyond single-model critique, recent systems explore broader forms of MLLM-assisted design evaluation, including heuristic-evaluation reproduction [11], [12], concrete UI-fix recommendation [13], multi-role critique [14], UX-metric selection [15], and feedback-based improvement of UI generation models [16]. However, these approaches mainly produce generic judgments or critiques, without modeling why a specific human response follows from the UI, question, and user context.

MLLM-as-Judge Benchmarks. A complementary line of work evaluates MLLMs as judges for UI/UX assessment. WiserUI-Bench [3] tests pairwise UI selection using A/B-tested variants with validated winners and expert interpretations, demonstrating that current MLLMs remain sensitive to position bias and unreliable at predicting behavior-aligned preferences. Similarly, MLLMJudge [4] compares MLLM judgments with human ratings of web interfaces. These benchmarks are valuable because they use human judgments or behavioral outcomes as the target signal, rather than relying only on expert heuristics or synthetic design defects. Our work also leverages human answers as supervision. PerceptUI builds on this human-centered principle, while explicitly supervising the model to predict how individual users respond and why.

LLM Agents as Synthetic Users. LLM agents have increasingly been used to approximate human behavior across domains [17]–[21]. Close to UI evaluation, SimUser [22] simulates mobile-app users to surface heuristic usability issues, UXAgent [23], [24] runs persona-conditioned agents on live web targets, and AgentA/B [25] deploys \(1{,}000\) agents on Amazon to approximate A/B testing outcomes. UXCascade [26] aggregates agent traces into usability findings, Avenir-UX [5] grounds interaction in GUI observations and produces SUS/SEQ-style reports, and SimAB [27] predicts outcomes for historical A/B experiments. Related work on LLM agents studies accessibility, reliability, and web interaction behavior [28]–[30]. OPeRA [31] records online-shopping sessions that pair user personas with observed interfaces, actions, and rationales, and [32] aligns LLMs with human cognitive walkthroughs.

Reasoning Distillation and Prompt Optimization. Our method also relates to reasoning distillation and prompt optimization. Prior work distills chain-of-thought rationales from stronger teachers into smaller students to improve reasoning [33], [34]. We adapt this idea to subjective UI/UX evaluation by supervising the model with contrastive rationales: each rationale explains why the recorded human answer fits the UI and persona better than the distractors. This follows the motivation of contrastive explanations, which clarify why one outcome occurred instead of another [35]. At inference time, we build on reflective prompt-optimization methods such as GEPA and TextGrad, which revise prompts from failure traces [36]. Our reflective prompt evolution applies this principle to UI/UX evaluation by updating the UI-grounding, persona-grounding using observed failures.

3 Method↩︎

3.1 Problem Formulation↩︎

We formulate UI/UX evaluation as persona-conditioned question answering. Each example is represented as a tuple \(x_i = (s_i, r_i, q_i, \mathcal{Y}_i, p_i)\), where \(s_i\) is a UI screenshot, \(r_i\) is an optional reference image such as an alternative icon or UI variant, \(q_i\) is a question, \(\mathcal{Y}_i=\{y_{i1},\ldots,y_{iK}\}\) is the set of answer options, and \(p_i\) is a natural-language persona describing participant \(i\). Given \(x_i\) and an instruction prompt \(P\), the model generates both a rationale \(z_i\) and an answer \(y_i\). The joint distribution can be formalized as: \(p_\theta(z_i, y_i \mid s_i, r_i, q_i, \mathcal{Y}_i, p_i; P)\), where \(\theta\) are the trainable parameters and \(P\) is task-specific prompt, with the answer marginal expressed as: \(p_\theta(y \mid \cdot) = \sum_{z} p_\theta(z, y \mid \cdot)\). We assume that the ground truth is the answer \(y^\star \in \mathcal{Y}\). Averaging the answer marginal over a population of personas \(\{p^{(n)}\}_{n=1}^{N}\), the model induces an answer distribution: \(\hat{p}(y \mid s, r, q; P) = \frac{1}{N}\sum_{n=1}^{N} p_\theta(y \mid s, r, q, \mathcal{Y}, p^{(n)}; P)\), which we can compare against the empirical distribution of human answers to evaluate a candidate design. In this work, \(p_\theta\) denotes the MLLM used for persona-conditioned answer prediction. To align \(p_\theta\) with genuine users, we adopt a two-stage pipeline: contrastive reflection fine-tuning teaches the model to disentangle human preferences, and reflective prompt evolution addresses residual errors.

3.2 Contrastive Reflection Fine-Tuning↩︎

A standard supervised objective trains the model to imitate the human answer \(y_i^\star\), but provides no direct supervision about why that answer is appropriate or why the alternatives are less suitable. This is limiting for subjective UI evaluation, where the correct response often depends on the interaction between visual evidence, question wording, and persona-specific preferences or constraints. Thus, we introduce contrastive reflection, a mechanism that enables agents to identify the reasons behind decisions and relate them to their internal context.

For each training example \(x_i=(s_i,r_i,q_i,\mathcal{Y}_i,p_i)\), we first construct task-conditioned auxiliary context. A UI grounding prompt extracts question-relevant visual evidence from the screenshot and optional reference image, yielding a summary \(u_i\). A persona grounding prompt maps the participant description to preferences, constraints, or sensitivities relevant to the question and answer options, producing a persona summary \(g_i\). Both summaries are generated without access to the ground-truth answer \(y_i^\star\) and are appended to the answer prompt. Conditioned on this augmented input and the recorded answer, a teacher model \(T\) generates a contrastive rationale: \[c_i^\star = T(s_i,r_i,q_i,\mathcal{Y}_i,p_i,u_i,g_i,y_i^\star).\] We structure \(c_i^\star\) into three short components: UI evidence, persona relevance, and option contrast. The persona-relevance component explains which aspects of the participant profile matter for the decision, if any. The option-contrast component justifies the recorded answer \(y_i^\star\) while contrasting it with each distractor \(y_{ik}\in\mathcal{Y}_i\setminus\{y_i^\star\}\).

Then, we fine-tune the student model \(p_\theta\) to generate the contrastive rationale followed by the recorded answer: \[{!}{ \mathcal{L}_{\mathrm{CRFT}}(\theta) = - \mathbb{E}_{(x_i,y_i^\star,c_i^\star)} \left[ \log p_\theta \left( c_i^\star, y_i^\star \mid s_i,r_i,q_i,\mathcal{Y}_i,p_i,u_i,g_i;P_{\mathrm{train}} \right) \right] } \label{eq:crft95loss}\tag{1}\] Unlike standard rationale distillation [34], which typically supervises a single positive reasoning trace, our target is answer-contrastive. The model is trained to justify the human answer relative to the full option set, encouraging UI-grounded and persona-aware comparisons rather than label imitation alone.

3.3 Reflective Prompt Evolution↩︎

After contrastive reflection fine-tuning, we employ the student model \(p_\theta\) as a synthetic user. Although contrastive reflection fine-tuning improves the model’s ability to compare UI evidence and persona-specific preferences, a generic prompt may generalize poorly across surveys and some residual errors may remain. We therefore optimize the prompt \(P\) using reflective prompt evolution. During this stage, the model parameters \(\theta\) remain fixed and only the prompt is optimized.

Starting from an initial prompt \(P_0\), each iteration evaluates the current prompt \(P_t\) on a minibatch \(\mathcal{B}_t \subset \mathcal{D}_{\mathrm{dev}}\). For each example, we collect the generated UI summary \(u_i\), persona summary \(g_i\), rationale \(c^{*}_i\), predicted answer \(\hat{y}_i\), and the corresponding human target. For population-level tasks, we aggregate predictions across personas to obtain the predicted answer distribution \(\hat{p}_t(\cdot \mid s_i,r_i,q_i,\mathcal{Y}_i)\). A prompted evaluator \(E\) compares this distribution with the empirical human distribution and produces a symbolic loss: \[\mathcal{L}_{\mathrm{sym}}^{(t)} = E(P_t,\mathcal{B}_t). \label{eq:rpe95loss}\tag{2}\] The symbolic loss records the signed residuals between predicted and empirical answer probabilities, together with representative failure traces. The screenshot and reference image are included in the evaluator input so that the critique remains grounded in the UI.

A second LLM analyzer \(A\) analyzes these failures and identifies weaknesses in the current prompt, such as over-reliance on generic persona traits, insufficient attention to the answer options, or weak use of the intermediate summaries. This produces a symbolic gradient: \[\nabla_{\mathrm{sym}}^{(t)} = A(P_t,\mathcal{L}_{\mathrm{sym}}^{(t)}), \label{eq:rpe95grad}\tag{3}\] which specifies prompt-level edits likely to reduce the observed residuals. A prompt optimizer \(O\) then applies these edits to produce a revised prompt: \[P'_t = O(P_t,\nabla_{\mathrm{sym}}^{(t)}). \label{eq:rpe95update}\tag{4}\] Each candidate prompt is evaluated on \(\mathcal{D}_{\mathrm{dev}}\) and added to a candidate pool. Before final selection, an LLM judge audits each candidate prompt and rejects prompts that contain answer leakage, dataset-specific shortcuts, or hard-coded decision rules. After \(T\) rounds, we return \(P^\star\), the candidate with the lowest mean residual on \(\mathcal{D}_{\text{dev}}\).

3.4 Backbone and Inference↩︎

We instantiate \(p_\theta\) with Qwen-VL. The model parameters \(\theta\) are updated only during contrastive reflection fine-tuning and are kept fixed during reflective prompt evolution. At inference, we employ the evolved prompt \(P\). Given an input \(x=(s,r,q,\mathcal{Y},p)\), the model first generates a UI grounding summary \(u\) and a persona grounding summary \(g\), and then predicts a rationale and answer: \[(\hat{z},\hat{y}) \sim p_\theta( \cdot \mid s,r,q,\mathcal{Y},p,u,g;P^\star ).\]

4 Experiments↩︎

Settings. We evaluate PerceptUI on six datasets covering complementary UI/UX tasks: WiserUI-Bench [3], UIClip/BetterApp [2], WebDevJudge [37], LabintheWild  [38], LabintheWild-UX  [39], and UICrit [7]. We additionally evaluate on a proprietary survey that captures how users perceive and interpret in-car interfaces, UXCar. For datasets without user profiles, we omit the persona field from the prompt and evaluate PerceptUI as a non-personalized UI evaluator.

Baselines. We compare PerceptUI against the baselines reported in each benchmark. For WiserUI-Bench, these include proprietary and open MLLMs, together with reasoning-enhanced methods such as CoCoT. For the remaining public benchmarks, we follow the original evaluation protocols and report the strongest baselines.

4.1 UI/UX Design Selection↩︎

A central task for UX research is selecting the best design. We therefore evaluate UI/UX selection on WiserUI-Bench [3], which contains 300 UI pairs with A/B-test-validated winners. Given two UI variants, the model is instructed to select the design more likely to guide users toward the intended action. We report First Accuracy (FA), Second Accuracy (SA), Average Accuracy (AA), and Consistent Accuracy (CA). CA is the most important metric as it requires the model to select the correct UI under both input orders, reducing the effect of position bias. Table 1 illustrates that existing MLLMs remain sensitive to input order. Several baselines obtain high SA but much lower FA, indicating a tendency to select the second image rather than consistently identify the A/B-test winner. PerceptUI improves the order-robust metrics, achieving the best AA and CA among all methods. The gain is most pronounced on CA, suggesting that PerceptUI better captures behavior-relevant UI evidence rather than relying on positional cues.

4.2 UI Design Quality↩︎

We next evaluate UI quality on the UIClip dataset  [2]. Unlike WiserUI-Bench, which tests whether a model can identify the A/B-test winner, UIClip/BetterApp focuses on visual design quality. Given two UI screenshots, the model selects the interface judged to have higher design quality. Table 2 depicts that zero-shot LVLMs perform close to chance, while UIClip variants benefit substantially from UI-specific training data. PerceptUI achieves the highest overall accuracy, with a modest gain over the strongest UIClip variant. This suggests that PerceptUI can act as a general UI evaluator even when persona information is unavailable, while also showing that specialized UI contrastive pre-training remains robust for static design-quality assessment.

4.3 Human Evaluation of UI/UX Rationales↩︎

Next, we conduct a human evaluation to assess whether model rationales are useful and grounded, beyond answer accuracy. Annotators are presented the UI, persona, question, answer options, predicted answer, and model explanation, and rate each rationale on UI grounding, persona use, contrastiveness, and overall usefulness. Table 3 shows that SFT produces the weakest explanations, indicating that label supervision alone does not teach the model to justify why an option fits the UI and persona. Models trained with contrastive rationales receive higher ratings, especially for distinguishing the selected answer from plausible alternatives. PerceptUI achieves the highest scores across all criteria, indicating that its rationales are more consistently UI-grounded, persona-aware, and informative about the predicted answer.

4.4 Persona-Conditioned UI Rating↩︎

[tbp]
\centering
\small
\resizebox{1.0\textwidth}{!}{
\begin{tabular}{llcccc}
\toprule
\textbf{Method} & \textbf{Setting} & \textbf{Acc.}$\uparrow$ & \textbf{MAE}$\downarrow$ & \textbf{JS Div.}$\downarrow$ & $\boldsymbol{\rho}$ $\uparrow$ \\
\midrule
Majority class & - & 24.13 & 1.92 & 0.282 & -- \\
\midrule
GPT-4o & Zero-shot & 27.84 & 1.42 & 0.184 & 0.412 \\
Claude Opus 4.6 & Zero-shot & 28.56 & 1.36 & 0.171 & 0.437 \\
Qwen2.5-VL-32B & Zero-shot & 26.09 & 1.51 & 0.198 & 0.391 \\
GPT-5.5 & Zero-shot & 29.45 & 1.22 & 0.150 & 0.429 \\
\midrule
No persona & SFT & 30.42 & 1.31 & 0.156 & 0.461 \\
Generic persona & SFT & 32.18 & 1.22 & 0.142 & 0.495 \\
\mathrm{\small PerceptUI} w/ generic persona & Generic persona & 35.46 & 1.16 & 0.135 & 0.521 \\
\mathrm{\small PerceptUI} w/o CR & Participant profile & 18.74 & 2.46 & 0.318 & 0.282 \\
\mathrm{\small PerceptUI} w/o RPE 
& Participant profile 
& \cellcolor{wiseryellow}\underline{41.23} 
& \cellcolor{wiseryellow}\underline{0.96} 
& \cellcolor{wiseryellow}\underline{0.103} 
& \cellcolor{wiseryellow}\underline{0.621} \\
\rowcolor{blue!8}
\mathrm{\small PerceptUI} 
& Participant profile 
& \cellcolor{wiserblue}\textbf{43.51} 
& \cellcolor{wiserblue}\textbf{0.88} 
& \cellcolor{wiserblue}\textbf{0.092} 
& \cellcolor{wiserblue}\textbf{0.658} \\
\bottomrule
\end{tabular}}
\caption{UI rating on LabintheWild. Accuracy and $\rho$ are higher better; MAE and JS divergence are lower better.}
\label{tab:reinecke}

Persona-conditioned UI rating is evaluated on the LabintheWild website-aesthetics dataset [38], where participants rate web screenshots on a \(1\)–\(9\) Likert scale and each rating is linked to participant profile information. Table [tab:reinecke] shows that zero-shot MLLMs improve only moderately over the majority class, while supervised baselines benefit from task-specific training. PerceptUI performs best across all metrics, improving exact-match accuracy, reducing rating error, and better matching empirical per-screenshot rating distributions. The gap between no-persona and persona-conditioned variants indicates that participant profiles provide a useful signal for subjective UI judgments.

4.5 Persona-Conditioned UX Perception↩︎

[tbp]
\centering
\small
\resizebox{1.0\textwidth}{!}{
\begin{tabular}{llcccc}
\toprule
\textbf{Method} & \textbf{Setting} & \textbf{Acc.}$\uparrow$ & \textbf{MAE}$\downarrow$ & \textbf{JS Div.}$\downarrow$ & $\boldsymbol{\rho}$ $\uparrow$ \\
\midrule
Majority class & - & 31.42 & 1.58 & 0.224 & -- \\
\midrule
GPT-4o & Zero-shot & 38.04 & 1.18 & 0.158 & 0.453 \\
Claude Opus 4.6 & Zero-shot & 39.81 & 1.12 & 0.146 & 0.476 \\
Gemini-2.5-pro & Zero-shot & 37.55 & 1.21 & 0.166 & 0.439 \\
GPT-5 & Zero-shot & 38.87 & 1.12 & 0.139 & 0.477 \\
\midrule
No persona & SFT & 42.07 & 1.07 & 0.131 & 0.501 \\
Generic persona & SFT & 44.18 & 1.01 & 0.118 & 0.534 \\
\mathrm{\small PerceptUI} w/ generic persona & Generic persona & 47.62 & 0.93 & 0.108 & 0.568 \\
\mathrm{\small PerceptUI} w/o CR & Participant profile & 10.83 & 2.86 & 0.494 & 0.235 \\
\mathrm{\small PerceptUI} w/o RPE
& Participant profile
& \cellcolor{wiseryellow}\underline{54.16}
& \cellcolor{wiseryellow}\underline{0.79}
& \cellcolor{wiseryellow}\underline{0.083}
& \cellcolor{wiseryellow}\underline{0.661} \\
\rowcolor{blue!8}
\mathrm{\small PerceptUI}
& Participant profile
& \cellcolor{wiserblue}\textbf{56.28} 
& \cellcolor{wiserblue}\textbf{0.71} 
& \cellcolor{wiserblue}\textbf{0.072} 
& \cellcolor{wiserblue}\textbf{0.703} \\
\bottomrule
\end{tabular}}
\caption{UX perception on LabintheWild-UX. }
\label{tab:miniukovich}

LabintheWild-UX extends persona-conditioned rating from visual appeal to broader UX perception. The dataset contains user ratings of UI screenshots across four website domains, with axes covering prototypicality, aesthetics, trustworthiness, and pre-use usability. This experiment evaluates whether PerceptUI can predict multiple aspects of perceived UX. We report metrics averaged across the six rating axes in Table [tab:miniukovich]. The results show that zero-shot MLLMs remain weaker than supervised models across these UX dimensions. Fine-tuning without persona information improves performance, but persona-aware variants are consistently stronger. PerceptUI achieves the best overall results, reducing rating error and better matching empirical response distributions. This highlights that participant-level context is useful for UX perception, where judgments of familiarity, trustworthiness, and expected usability depend partly on user expectations.

4.6 UX Question Answering for Automotive Interfaces↩︎

We next examine whether PerceptUI transfers beyond web and mobile interfaces to automotive UX scenarios. UXCar contains participant responses to questions about vehicle-interface screenshots, where answers depend on visual cues, interface conventions, and familiarity with automotive displays. Questions are divided into four groups: comprehension, clarity, perception, and spatial location. Figure 2 reports macro-F1 for each group. Questions about action and interface clarity are generally easier, as they rely on more explicit screen cues. Perception and spatial location are more challenging as they require attention to specific visual elements and may depend on participants’ familiarity with in-car interfaces. PerceptUI achieves the strongest performance across all groups. The drop without contrastive reflection further suggests that explaining why one answer fits better than the alternatives helps the model capture user-specific judgment cues beyond standard answer imitation.

5 Conclusion↩︎

We introduced PerceptUI, a framework for automated UI/UX evaluation. By combining contrastive reflection fine-tuning with reflective prompt evolution, PerceptUI predicts how a user would answer UI/UX questions and explains the prediction relative to alternative answers. Experiments demonstrate improvements in both personalized and non-personalized UI/UX judgments across diverse tasks, including design selection, rating prediction, population-level response estimation, and critique generation. Additional analyses show that strong performance depends not only on the underlying vision-language model, but also on grounding predictions in user context, contrasting alternative answers, and making the model’s internal evidence explicit through structured explanations. These results highlight PerceptUI as a promising foundation for research and industry applications at the intersection of user modeling, interface evaluation, and human-centered product development.

6 Limitations↩︎

Despite achieving strong performance across our evaluations, several limitations should be acknowledged. First, the reproducibility of some experiments is limited because our proprietary dataset is not publicly available. Second, PerceptUI may inherit biases from the underlying vision-language model and from the data used to construct personas, including cultural, gender, age, accessibility, and socioeconomic biases. This is especially important because the framework predicts user responses and could otherwise over-represent the preferences of groups that are more visible in the training data.

Another limitation is that UI/UX evaluation is inherently interactive. Many usability problems only emerge through multi-step interaction, such as navigation breakdowns, delayed feedback, error recovery, form completion, or mismatches between user expectations and system responses. PerceptUI operates on screenshots and evaluation questions, and therefore cannot fully capture the temporal and procedural aspects of user experience. While the WebDevJudge results demonstrate that the framework can transfer to generated web interfaces in an image-only setting, modeling interaction traces, browser actions, and task-completion outcomes remains an important direction for future work.

In addition, the generated rationales should also be interpreted with care. Although contrastive reflection improves rationale quality, these explanations are model-generated and should not be treated as the true causal reasons behind a participant’s response.

Another limitation is that the set of baselines varies across experiments. This is because the benchmarks differ substantially in task format, available inputs, and evaluation protocol: some focus on pairwise UI selection, others on rating prediction, explanation recovery, or open-ended critique. As a result, not all prior methods are applicable to every setting, and some benchmarks provide only domain-specific baselines. We therefore compare against the strongest available and compatible baselines for each task, but the results should be viewed as task-specific comparisons rather than a single unified leaderboard across all benchmarks.

Finally, PerceptUI depends on the quality of the teacher model, the base vision-language model, and the available persona information. Errors in these components can lead to incorrect predictions, overconfident population estimates, or plausible but weakly supported rationales. Our ablations examine these factors, but further work is needed on reproducibility, bias auditing, dynamic interaction modeling, and human validation.

7 Ethics Statement↩︎

This paper introduces an MLLM-based framework for persona-conditioned UI evaluation, enabling early-stage assessment of how different users may perceive and respond to interfaces. While this can support faster and more targeted design iteration, it also raises ethical considerations.

First, PerceptUI may reproduce or amplify biases present in the underlying vision-language model, the participant data, or the persona representation. Predicted responses may therefore reflect stereotypes about age, gender, culture, disability, digital literacy, occupation, or socioeconomic status rather than the views of real users. This is particularly important when the system is used to compare designs for different user groups, since biased predictions could lead to interface decisions that benefit some groups while disadvantaging others.

Second, simulated user responses should not be treated as a replacement for human participation. Although synthetic evaluation can reduce the cost of early exploration, relying on it too heavily may marginalize the voices of actual users, especially users from underrepresented or accessibility-sensitive groups. There is also a risk that user-conditioned predictions could be used to optimize interfaces for persuasion or behavioral manipulation rather than usability, transparency, or user well-being. Such applications require careful oversight, clear documentation, and human review.

Finally, participant-linked UI data can contain sensitive information about preferences, habits, accessibility needs, or decision-making patterns. Any deployment of PerceptUI should therefore follow appropriate privacy safeguards, limit unnecessary data retention, and avoid using inferred personas to make consequential decisions about individuals. We view PerceptUI as a tool to complement, not replace, human-centered design practices. By using synthetic evaluation as an early diagnostic signal while preserving human validation in product decisions, we aim to support UI evaluation that is transparent, inclusive, and socially responsible.

8 Experimental Setup↩︎

Different benchmarks provide different forms of supervision. We convert all benchmarks into the same input-output interface used by PerceptUI: a UI screenshot or screenshot pair, an evaluation question, a set of answer options, and an optional persona. When participant-level profiles are available, they are included as persona context. When no participant profile is provided, the persona field is omitted, and the model is evaluated as a non-personalized UI evaluator. For pairwise preference benchmarks, the answer options correspond to the two UI variants. For rating benchmarks, the answer options correspond to the discrete rating scale. For explanation benchmarks, the model is prompted to generate a rationale or critique conditioned on the benchmark-specific target format. Unless otherwise stated, we use GPT-5.5 as the teacher model for generating contrastive rationales during contrastive reflection fine-tuning (Figure 3).

To avoid leakage between model training and prompt selection, we use disjoint data for contrastive reflection fine-tuning and reflective prompt evolution. For each dataset, examples are first split into non-overlapping training, development, and test partitions. The training partition is used only for contrastive reflection fine-tuning: teacher rationales, question paraphrases, UI grounding summaries, and persona grounding summaries are generated only for training examples, and only these examples are used to update the student parameters \(\theta\). The development samples are used only for reflective prompt evolution. During this stage, \(\theta\) is fixed, and no gradient update is performed. Development examples are used to evaluate prompt candidates, compute symbolic feedback, and select the final prompt \(P^\star\), but they are never used to fine-tune the model or to generate supervised training targets for CRFT. The test partition is held out from both stages and is used only for final evaluation. This separation ensures that the model is not fine-tuned on the same examples used to optimize prompts, and that reported test performance does not rely on examples seen during either parameter training or prompt evolution.

After each prompt update during prompt evolution (see Figure 4), we run an additional LLM-based validation step before adding the candidate prompt to the pool. The judge verifies that the revised prompt does not contain gold labels, empirical answer distributions, dataset-specific shortcuts, example identifiers, or rules that directly encode development-set failures. It also checks that the update does not introduce overly specific instructions tied to particular screenshots, personas, questions, or answer options. Prompts that fail this check are rejected. In addition, the prompt optimizer is explicitly instructed to avoid leakage, memorization, dataset-specific heuristics, and narrow rules. Instead, it must propose general edits to the decision procedure, such as improving the use of UI evidence, answer-option comparison, or persona-question grounding.

To make the training and prompt-optimization procedure reproducible, we summarize the main implementation settings in Table 4. These include the student model, fine-tuning configuration, teacher-generation settings, reflective prompt evolution parameters, and other relevant parameters.

For LabintheWild and LabintheWild-UX datasets (Tab. [tab:reinecke], Tab. [tab:miniukovich]), we evaluate both individual rating prediction and aggregate alignment. Accuracy measures exact agreement with the recorded rating, while MAE measures the average distance from the target rating and therefore accounts for near misses on fine-grained scales. JS divergence compares predicted and empirical rating distributions for each screenshot, and Spearman \(\rho\) measures whether the model ranks interfaces in the same order as human ratings. Since the majority-class baseline assigns the same prediction to all examples, its rank correlation is undefined. For the UXcar analysis (Fig. 2), each agent predicts the answer of a matched participant. We therefore report macro-F1, which averages performance across answer options and reduces the influence of dominant responses. The overall score is computed as the mean across questions and corresponds to the macro-F1 reported in Tab. 10. We report reasoning-enhanced prompting baselines, including CoCoT, Self-Refine, DDCoT, and multi-agent debate, on WiserUI-Bench following its evaluation protocol. We do not apply these baselines to all pairwise-selection benchmarks because their relevance depends on the task format and metrics. In particular, multi-agent debate is mainly designed to reduce order-dependent errors, which WiserUI-Bench captures through FA, SA, and Consistent Accuracy. UIClip/BetterApp, in contrast, reports a single overall-accuracy metric and focuses on visual design-quality preference rather than A/B-test outcome prediction. Its dominant challenge is therefore perceptual grounding of design quality, rather than inconsistent reasoning across input orders. We consequently report reasoning-enhanced prompting baselines where they are diagnostically informative, and use the strongest task-appropriate baselines for the remaining benchmarks.

For the human evaluation in Section 4.3, we sample 120 UI–persona–question instances from the test split. For each instance, annotators are shown the UI screenshot, persona, question, answer options, predicted answer, and one anonymized rationale. Each rationale is evaluated by three annotators along four dimensions: UI grounding, persona use, contrastiveness, and overall usefulness. Ratings are collected on a 1–5 Likert scale. Model identities are hidden, and rationale order is randomized. We report mean scores across annotators and examples.

9 Discussion↩︎

Experimental results suggest that persona-conditioned UI/UX evaluation offers a useful alternative to treating MLLMs as generic interface judges. Across tasks, PerceptUI is strongest when the model must connect visual evidence, answer options, and user-specific context. This is consistent with the motivation for contrastive reflection: subjective UI judgments often depend not only on why one answer is plausible, but also on why nearby alternatives are less appropriate for a given user.

At the same time, PerceptUI should be viewed as an early-stage evaluation tool rather than a replacement for human studies or online A/B tests. Its predictions can help identify likely usability issues, compare design alternatives, and prioritize interfaces for further testing, but final product decisions should still require validation with real users. This distinction is important because the model can approximate patterns in recorded responses, but it cannot fully observe the motivations, expectations, or situational factors that shape an individual user’s judgment.

Our experiments also highlight the importance of persona quality. Generic or mismatched personas provide limited benefit, while participant-level information improves prediction and calibration. This suggests that persona-conditioned evaluation depends not only on whether user information is included, but also on whether the representation captures factors relevant to the UI question. Richer personas may improve fidelity, but they also require stronger privacy safeguards and more careful validation.

An important implication of our results is that individual-level prediction and population-level calibration should be interpreted separately. PerceptUI may still make mistakes for a particular user, especially when the available persona omits relevant context. However, aggregating predictions across many profiles can produce useful estimates of population-level response distributions. This makes the framework more appropriate for early design screening, subgroup analysis, and prioritizing follow-up studies than for making definitive claims about any single user.

Finally, the available benchmarks considered in this work cover complementary but incomplete views of UI/UX quality, including preference prediction, design quality, generated web interfaces, critique quality, and UX perception. Future work should extend UI/UX evaluation to interaction with the interface and longer-term user histories, so that models can better capture how users perceive, navigate, and adapt to interfaces over time.

10 Cost Analysis↩︎

We report the cost of PerceptUI for evaluating one UI with five questions and 20 personas. Frontier models tend to be inexpensive for a single judgment, but become costly when each UI must be evaluated separately for multiple personas. PerceptUI shifts part of the computation to an offline stage, where contrastive rationales are generated, and the student model is fine-tuned. Once trained, UI evaluation can be performed with a smaller hosted model or a self-hosted VLM. Table 5 shows that frontier-model evaluation grows linearly with the number of sampled personas, since each persona requires a separate query. In contrast, PerceptUI incurs a one-time preparation cost but substantially reduces the per-UI inference cost. This makes large-scale evaluation of UI variants more practical than repeatedly querying a frontier model for each simulated user.

11 Prompts↩︎

We provide simplified prompt templates used by PerceptUI. The full prompts include dataset-specific formatting instructions and output constraints, which are omitted for readability.

11.1 UI Grounding Prompt↩︎

The UI grounding prompt produces the visual summary \(u_i\) used by the answer simulator and, during training, by the contrastive reflection teacher.

11.2 Persona Grounding Prompt↩︎

The persona grounding prompt produces the persona summary \(g_i\) used by the answer simulator and contrastive reflection teacher.

11.3 Persona-Conditioned Answer Simulation↩︎

At inference, the answer simulator predicts a rationale \(\hat{z}_i\) and an answer \(\hat{y}_i\) from the UI grounding summary \(u_i\), persona grounding summary \(g_i\), question, and answer options.

Multiple-choice questions.

Location-grid questions.

Likert-scale questions.

11.4 Contrastive Reflection Teacher↩︎

For each training example \((x_i, y_i^\star)\), the teacher \(T\) is prompted with the inputs of \(x_i\), the UI grounding summary \(u_i\), the persona grounding summary \(g_i\), and the recorded human answer \(y_i^\star\). The teacher produces the contrastive reflection \(c_i^\star\) used as the CRFT supervisory target.

11.5 Reflective Prompt Evolution↩︎

Reflective prompt evolution updates the prompt configuration \(P=(P_{\mathrm{ui}},P_{\mathrm{persona}},P_{\mathrm{ans}})\) while keeping the model parameters fixed. We provide simplified versions of the evaluator, analyzer, optimizer, and audit prompts.

Failure evaluator.

Prompt analyzer.

Prompt optimizer.

Prompt audit.

12 Datasets↩︎

Table 6 summarizes whether each dataset provides participant-level profile information. When profiles are available, we include them as persona context. When profiles are unavailable, we omit the persona field and evaluate PerceptUI as a non-personalized UI evaluator.

12.0.0.1 UXCar Dataset.

We leverage a proprietary participant-level UI/UX evaluation dataset, denoted as UXCar. This benchmark focuses on in-vehicle interface evaluation and contains approximately \(500\) participants and \(30\) UI/UX questions. Each example consists of a UI screenshot, a participant profile, an evaluation question, and a multiple-choice answer. Participant profiles contain user attributes such as age and gender. Questions cover several aspects of UI/UX perception, including interface understanding, spatial preference, icon interpretation, and perceived usability. Some questions additionally provide a reference icon or auxiliary visual cue together with the main interface screenshot. For example, one question presents a navigation or map interface and asks whether the participant correctly understands a driving instruction, such as where or when to turn. The dataset is designed to capture subjective user judgments conditioned on both visual interface content and participant characteristics, enabling evaluation of persona-conditioned UI response prediction and population-level calibration.

13 Additional Experiments↩︎

13.1 Rationale Alignment in UI/UX Selection↩︎

WiserUI-Bench also provides expert-written rationales for the A/B-test winner in each UI pair. This allows us to test whether model explanations recover the design factors experts identify as supporting the more effective variant. Following the benchmark protocol, the winning UI is given, and the model generates an explanation for the preference. As shown in Table 7, prior methods recover only part of the expert rationale space. PerceptUI achieves the highest interpretation-level and instance-level recall, indicating stronger alignment with human UI/UX interpretations. These results further show that the agent can associate design preferences with concrete UI factors.

13.2 Quality of UI Critiques↩︎

Beyond selecting preferred designs or identifying relevant design principles, UI/UX evaluation also requires actionable feedback that tells designers what to improve and where. UICrit [7] provides a suitable benchmark for evaluating UI critique quality, as it contains expert-written critiques for mobile UI screenshots, with comments linked to relevant screen regions and design-quality ratings. We compare automatic feedback against zero-shot Gemini, the strongest few-shot visual-prompting baseline, and human designer critiques. Table 8 suggests that human designer critiques remain stronger than all LLM methods, highlighting the difficulty of open-ended UI feedback. It further indicates that our model produces more useful critiques than other baselines, despite using a substantially smaller model than frontier proprietary MLLMs. This suggests that few-shot imitation of critique examples is insufficient for producing grounded feedback, often leading to shallow comments that do not capture why a design issue matters.

13.3 Web Interface Preference↩︎

With the rise of AI-generated interfaces, UX evaluation increasingly involves comparing alternative web prototypes produced from the same user request. We therefore evaluate on WebDevJudge [37], which measures preferences between generated web implementations. This setting is less standardized than WiserUI-Bench or UIClip as interfaces may differ not only in visual design, but also in layout, content organization, and how well they satisfy the requested task. Given two UI screenshots, the model is prompted to select the preferred design. Figure 5 reveals that agreement remains low for all image-only judges, with the strongest baselines differing by only a few points. PerceptUI obtains the best agreement, suggesting that its UI judgment transfers to less standardized, generated interfaces. The modest improvement further indicates that some preferences depend on task satisfaction, interaction flow, or implementation details that cannot be fully inferred from a static screenshot.

13.4 UI Preference Explanation↩︎

In practice, UX researchers and designers need more than a preference label. They also need to know which design principles explain why one interface is better than another. We assess this task on UIClip/BetterApp [2]. Given a pair of UI screenshots and the preferred design, the agent predicts which CRAP principles (contrast, repetition, alignment, and proximity) explain the preference. This task measures whether the model can connect design-relevant evidence behind a choice, rather than simply selecting a higher-quality UI. Following UIClip, we report the macro-averaged F1 over the four CRAP principles, together with a choice-adjusted F1 that ignores a model’s suggestions when it selects the wrong preferred UI. Figure  6 illustrates that frontier MLLMs achieve competitive F1 but much lower choice-adjusted F1, consistent with their tendency to mention many plausible principles even when the selected UI is incorrect. PerceptUI achieves the strongest F1 under both metrics. This suggests that its suggestions are better tied to the correct design preference, rather than reflecting generic principle coverage.

13.5 Population-Level Alignment↩︎

Beyond predicting individual answers, PerceptUI aims to estimate how a population would respond to a UI/UX question by aggregating persona-conditioned predictions. Figure 7 evaluates this distributional calibration on UXcar, by aggregating answers over \(N\) agents. The predicted answer probabilities are close to the observed human frequencies across answer options, although the model is slightly overconfident when assigning very high probabilities. The right figure also shows the effect of the persona budget \(N\). We compare two aggregation methods. In hard voting, each persona contributes only its most likely answer. In soft aggregation, each persona contributes its full predicted answer distribution. Soft aggregation produces lower JS divergence because it preserves uncertainty from each persona instead of collapsing every prediction to a single choice. Increasing \(N\) improves calibration, with gains beginning to saturate around \(N{\approx}64\).

13.6 Sensitivity to the Teacher Model↩︎

Contrastive reflection fine-tuning depends on rationales generated by a teacher model, so we test whether the gains are tied to the default teacher. We replace GPT-5.5 with four alternative teachers and keep the student model, training data, and prompt-evolution procedure fixed. Figure 8 shows that frontier teachers yield similar downstream accuracy and rationale quality, with differences close to seed-level variation. This suggests that PerceptUI is relatively robust to the teacher choice. Performance drops when using smaller or weaker teachers, including Claude 4.5 Haiku and Qwen2.5-VL-72B, indicating that the quality of the generated contrastive rationales still matters. Overall, the results support the design of CRFT while showing that teacher choice affects the quality of the supervision signal.

13.7 Generalization to Unseen Questions and Personas↩︎

This study assesses whether PerceptUI learns reusable user–UI judgment patterns rather than memorizing survey-specific questions or participant identities. We test three generalization settings: unseen questions, unseen participants, and the combination of both, allowing us to evaluate generalization to new question formulations and unseen user personas. As illustrated in Table 9, as expected, performance drops for all methods, with the largest degradation when both questions and participants are unseen. PerceptUI remains the strongest method across all settings, indicating that its gains are not limited to memorizing familiar users or question forms. The improvement is especially noticeable for unseen questions, where the model must interpret new wording while grounding its prediction in the UI and persona. On the other hand, the SFT baselines degrade more sharply, as imitating answers teaches the model to reproduce recorded answers without understanding why a choice fits a given user and UI.

13.8 Visual Evidence Localization↩︎

Grounded UI critique requires not only identifying a design issue, but also locating the visual evidence that supports it. We evaluate this ability on UICrit [7], where the model outputs a bounding box for the UI region associated with its critique or predicted answer. Figure 9 indicates that localization remains challenging for all methods. The strongest UICrit baseline relies on image patches and achieves the highest IoU among the original baselines. PerceptUI improves over coordinate prompting, but remains slightly below the patch-based baseline, which is expected because our model is optimized for answer prediction and explanation rather than dense visual grounding. The drop without contrastive reflection suggests that contrastive rationales also help the model attend to UI regions that distinguish the selected answer from alternatives.

13.9 Evolution of Prompts↩︎

Reflective prompt evolution is intended to adapt the inference prompt to recurring errors without changing the model parameters. To examine this process, each run starts from the same initial task-description prompt, and performance is tracked across optimization rounds. Figure 10 indicates that most gains occur in the early iterations, followed by more gradual improvements. This suggests that early rewrites address broad prompt issues, such as ambiguous scale interpretation, weak use of persona information, or insufficient comparison between answer options. Later revisions mainly correct narrower errors and yield smaller marginal gains. The amount of improvement varies across benchmarks. The participant survey and UIClip/BetterApp benefit most, WiserUI-Bench improves more moderately, and WebDevJudge changes least. This pattern is consistent with the role of prompt evolution: it is most effective when errors arise from task framing or interpretation, and less effective when the relevant evidence is not fully available in the input.

13.10 Ablation Study↩︎

We ablate the main components of PerceptUI on UXcar. We first examine the role of persona information by removing the persona, replacing it with a shuffled participant profile, or using a shared generic description. We then isolate the effect of the learning and inference stages by comparing answer-only fine-tuning (SFT), contrastive reflection fine-tuning, and reflective prompt evolution. Table 10 shows that persona variants provide only limited gains when the profile is generic or mismatched, suggesting that participant-level prediction requires user context that is both specific and correctly grounded. Answer-only fine-tuning also remains limited, as it learns from target labels without modeling why an answer is preferred over alternatives. In contrast, contrastive reflection fine-tuning yields the largest improvement, indicating that rationales grounded in the UI, persona, and answer options provide useful supervision for learning user-specific judgment cues.

14 Examples↩︎

References↩︎