1 Introduction↩︎

Large Language Models (LLMs) are increasingly used as judges for model comparison, reward modeling, and benchmark construction because they offer a scalable alternative to human evaluation [1]–[4]. However, most LLM-as-Judge pipelines target an aggregate signal: models are trained, aligned, or evaluated against pooled preferences that collapse across annotators [5]–[8], so they simulate crowd-level consensus rather than any specific evaluator. This is a key limitation for open-ended evaluation, where a single shared standard may not exist and annotators apply different criteria to the same outputs [9], [10]. In such settings disagreement is often meaningful rather than noise [11]–[13]. Even LLM judges that match average ratings can miss group- and person-level differences [14]. Current models struggle to infer individual preferences from value-laden statements [15]. Matching consensus thus does not guarantee faithful simulation of any individual evaluator.

This motivates a different target: individual evaluator simulation, which can support evaluation auditing, per-person reward modeling, and fairness analysis of whose perspectives evaluation pipelines underrepresent. Prior personalized methods condition on persona descriptions or compact preference profiles [16], [17], but represent people through summaries rather than their own decision traces. We instead ask: given an evaluator’s historical judgments and decision traces, can an LLM simulate how that evaluator will judge a new instance?

We argue that this individual evaluator simulation requires richer supervision than final labels alone. A final label captures an outcome, but not the process that produced it. We therefore propose PersonaJudge (1), an evaluator-specific framework that uses multi-facet evaluator data: (1) a primary categorical judgment, (2) interface telemetry capturing how evaluators inspect the task, and (3) retrospective reasoning giving post-hoc explanations of why decisions were made.

Using this framework, we evaluate whether evaluator-specific data can improve simulation over the Base Judge, which complementary signals contribute most, and how performance varies across evaluators. We conduct a systematic empirical study with data from 32 trained annotators and 4,200 preference judgments in a factorial design. Extensive experiments show that PersonaJudge improves simulation accuracy over the Base Judge by up to 9.9 points; retrospective reasoning gives the largest gains, while interface telemetry often reduces performance. We also find that simulation difficulty is systematic—predicted by an evaluator’s neutral usage and divergence from consensus—with neutral usage a stable cross-task trait.

In summary, our contributions are:

• We propose a multi-facet simulation framework PersonaJudge for evaluator-specific judgment behavior using categorical judgments, interface telemetry, and retrospective reasoning.

• We conduct a systematic empirical analysis of how complementary evaluator-specific signals, model choice, and demonstration count affect the simulation performance.

• We characterize evaluator-level simulatability, showing that simulation difficulty varies widely but systematically across evaluators—predicted by their neutral usage and divergence from consensus—and that neutral usage is a stable, cross-task trait.

2 Related Work↩︎

We review three relevant lines of research: LLM-as-Judge, individualized alignment, and process-oriented behavioral/cognitive traces for modeling human decision behavior.

2.0.0.1 Language models as judges

LLMs are increasingly used as evaluators to simulate human judgments. LLM-as-a-Judge methods have shown strong alignment with aggregate human judgments, including improved correlations with human ratings and high agreement with human preferences [2], [3] and have rapidly become a standard evaluation paradigm [18], [19], but they rely on standardized rubrics that obscure individual variation in judgment styles and criteria application. Studies have highlighted concerns about the biases and robustness in these methods [3], [20]–[22]. Moreover, the reproducibility of LLM-based evaluation has been questioned because model behavior can drift across updates and evaluation criteria may evolve during the validation process [23], [24].

Traditional crowdsourcing approaches similarly aggregate multiple annotations to improve reliability, but such aggregation can obscure meaningful differences in annotator judgments and consistency [25], [26]. Popular preference datasets and pipelines are often formulated around binary pairwise comparisons [5], [7], [27], [28], which can make it difficult to represent neutral, uncertain, or mixed judgments that arise in realistic scenarios involving underspecified criteria.

2.0.0.2 Individualized alignment

Beyond using LLMs as judges during evaluation, a related line of work examines how human preferences are modeled in alignment pipelines. These approaches have improved models’ ability to align with human feedback, but they largely treat preferences as outcome labels and rarely model how individual evaluators arrive at their decisions. Reinforcement Learning from Human Feedback approaches [6], [7] train reward models on aggregated human preference data using techniques like Bradley-Terry modeling [29]. Later works such as Constitutional AI [1] and RLAIF [30] extend this paradigm by incorporating AI feedback, but they aim to learn a single, static preference function. Recent multi-objective and diversity-aware alignment methods model heterogeneous preferences through Pareto trade-offs or mixtures of reward models, but they remain formulated at the population level rather than simulating a particular evaluator [31], [32]. Persona-based conditioning can be unreliable, because available persona descriptions can be too sparse or simplistic to predict task-specific preferences [16]. Even when users attempt to guide an LLM more directly, they may struggle to translate their implicit standards into explicit labeling instructions: iterative prompt refinement without gold labels does not reliably improve labeling accuracy [33]. Other work models or simulates individual responses by conditioning on self-reports—interviews and surveys [34] or persona and demographics [35]—but relies on elicited descriptions of a person rather than an evaluator’s own decision traces for preference judgment. These limitations suggest that evaluator-specific simulation requires richer supervision beyond preference outcomes, including behavioral traces of how evaluators inspect evidence and explanatory traces of how they justify their decisions.

2.0.0.3 Behavioral and cognitive traces

Prior work on process-oriented supervision points to two signals that may complement outcome-only preference modeling.

Behavior-wise, recent work has begun to model people through their interaction traces rather than only their final outputs. For example, [36] collect computer-use observations to build general user models from everyday interactions, showing that behavioral traces can support inference about users’ preferences. [37]‘s OPeRA dataset goes further by collecting browser observations, fine-grained web actions, and self-reported rationales to benchmark personalized human behavior simulation in online shopping. These studies suggest that interaction traces can expose how users gather evidence and act in context, but they focus on user modeling or next-action prediction rather than open-ended judgment simulation. Most closely, [38] augment preference judgments with annotators’ mouse-tracking reading traces, but analyze the dataset rather than simulating annotators, and use study-time traces rather than deployment telemetry with retrospective reasoning.

Cognitive-wise, a separate line of work shows the value of natural language explanations and feedback for making human reasoning more legible to learning systems. Chain-of-Thought prompting and related evaluation methods show that intermediate reasoning can improve model performance and judgment quality [2], [39], [40], but they rely on model-generated reasoning rather than human-authored judgment explanations. More broadly, prior work shows that human-provided explanations and feedback can support model development [41], [42]. However, these efforts are not aimed at simulating how a particular evaluator makes subjective judgments in open-ended comparison settings.

Existing work has explored richer evaluator-specific signals, but has not examined whether behavioral and cognitive traces jointly support evaluator-specific judgment simulation. Our work addresses this gap using interface telemetry and retrospective reasoning as complementary supervision.

3 Simulating Individual Preference Judgments with PersonaJudge↩︎

We introduce PersonaJudge, a novel approach for simulating individuals’ preference judgments using LLMs and their ability to perform in-context learning (ICL) [43] with multi-facet, evaluator-specific demonstration data. Unlike consensus-based preference modeling [6], [7], PersonaJudge simulates individual evaluators’ decision-making processes by using a primary categorical judgment signal, interface telemetry, and retrospective reasoning.

3.1 Problem Formulation↩︎

3.1.0.1 Setup

Given a target evaluator \(e_j\) and a target evaluation instance \(x=(c,r^A,r^B)\), where \(c\) is the conversation context and \(r^A,r^B\) are the two candidate responses, PersonaJudge simulates \(e_j\)’s three-class judgment \(y\in\{A,N,B\}\) (prefer A, neutral, prefer B) through in-context learning over evaluator-specific demonstrations. Let \(x_i=(c_i,r_i^A,r_i^B)\) denote a prior instance previously judged by \(e_j\) with label \(y_{i,j}\in\{A,N,B\}\) and an optional explanatory signal \(s_{i,j}\) (interface telemetry and/or retrospective reasoning).

3.1.0.2 Two-round elicitation

We elicit the judgment as two binary decisions. A preference-detection step first determines whether \(e_j\) expresses a preference; a direction step, invoked only when a preference is detected, then determines which option is preferred. Define preference label \(b(y)=\mathrm{\small neutral}\) if \(y=N\) and \(b(y)=\mathrm{\small pref}\) otherwise, and the direction \(\delta(y)=y\) for \(y\in\{A,B\}\). Both rounds use the same prompt structure—an instruction header, \(k\) evaluator-specific demonstrations, and the unlabeled query \(\mathrm{Q}(x)\)—but differ in their round-specific label mapping and sampling constraints: Round 1 maps labels to preference vs.no preference, whereas Round 2 uses directional preference labels only (Appendix [sim-prompt]).

3.1.0.3 Round 1 (preference detection)

PersonaJudge constructs the prompt \[\mathrm{P}^{1}(e_j,x)=\mathrm{Intro}^{1}\oplus\bigoplus_{i=1}^{k} \mathrm{Demo}\!\big(x_i,\,b(y_{i,j}),\,s_{i,j}\big)\oplus \mathrm{Q}(x), \notag\] where each demonstration carries the binary label \(b(y_{i,j})\in\{\mathrm{\small neutral},\mathrm{\small pref}\}\). The model returns \(\hat{p}\in\{\mathrm{\small neutral},\mathrm{\small pref}\}\).

3.1.0.4 Round 2 (direction)

If \(\hat{p}=\mathrm{\small pref}\), PersonaJudge constructs \[\mathrm{P}^{2}(e_j,x)=\mathrm{Intro}^{2}\oplus \!\!\bigoplus_{i:\,y_{i,j}\neq N}\!\! \mathrm{Demo}\!\big(x_i,\,\delta(y_{i,j}),\,s_{i,j}\big)\oplus \mathrm{Q}(x), \notag\] using demonstrations sampled from the same evaluator-specific pool, restricted to instances in which \(e_j\) expressed a directional preference, each labeled with its direction \(\delta(y_{i,j})\in\{A,B\}\). The model returns \(\hat{d}\in\{A,B\}\).

3.1.0.5 Composition

The simulated judgment recovers the full three-class space, \[\hat{y}= \begin{cases} N & \text{if } \hat{p}=\mathrm{\small neutral},\\ \hat{d} & \text{if } \hat{p}=\mathrm{\small pref}, \notag \end{cases}\] making the neutral decision explicit while keeping each round a binary choice. In the prompts, the two outcomes of each round are encoded as the rating numbers \(1\) and \(2\) (Appendix [sim-prompt]).

3.2 Demonstration Data Model↩︎

PersonaJudge demonstrations combine categorical preference labels {Prefer A, Neutral, Prefer B} with two types of explanatory data: interface telemetry and retrospective reasoning. These two signals complement categorical judgments in distinct ways: Interface telemetry provides an implicit trace of how evaluators inspect tasks, while retrospective reasoning provides an explicit account of the criteria and trade-offs behind their decisions. Unlike binary preference datasets [5], [7], PersonaJudge keeps Neutral as a valid third label instead of collapsing or removing neutral judgments. This three-class judgment is the canonical stored demonstration label; round-specific binary labels are derived from it only at prompt construction time.

3.2.1 Interface Telemetry Data↩︎

Interface telemetry augments categorical judgments with implicit behavioral traces of how evaluators inspect information during decision making—what was inspected, in what order, for how long, and whether regions were revisited [44], [45]. Process-tracing research shows that decision processes leave informative behavioral traces [46], [47], and interaction data such as clicks and dwell time can help infer user goals and information access behavior [48], [49]. Incorporating these traces lets PersonaJudge condition on not only what an evaluator decided, but also how they inspected the task before deciding.

3.2.2 Retrospective Reasoning Data↩︎

Retrospective verbal reports can complement categorical judgments by providing explicit accounts of the evaluator’s decision process [50]: a final label alone does not reveal which aspects of the candidate responses an evaluator prioritized, nor why they selected Neutral when responses satisfied different criteria. Such natural-language explanations can make decision-relevant considerations more explicit in a form language models can directly use [51], [52], surfacing evaluator-specific judgment criteria that labels alone do not. Because these reports are collected after task completion, they avoid interfering with the judgment itself [53], [54]. In our setting, this signal is especially useful for ICL-based simulation because it provides evaluator-specific judgment criteria beyond labels alone. We operationalize this signal as post-task think-aloud transcripts.

4 Study Design↩︎

We conduct an empirical study to evaluate the effectiveness of PersonaJudge in simulating individual human judgments to address the research questions (RQs) introduced in 5.

4.1 Evaluation Task Setup↩︎

We sample instances from Anthropic’s Helpful and Harmless (HH) dataset [7], a widely adopted, publicly available preference benchmark whose two distinct evaluation criteria—helpfulness and harmlessness—engage different evaluator reasoning and enable our cross-task analysis. We select 700 conversations each for the helpfulness and harmlessness evaluation tasks. We recruited 32 trained annotators, 10 of whom contributed to both datasets, resulting in 21 evaluators per dataset. Each evaluator completed 100 judgments per dataset, yielding 2,100 judgments for helpfulness and 2,100 for harmlessness. We randomize the presentation order and implement position-bias controls by varying the response ordering across evaluators [3], [20].

4.2 Multi-Facet Data Collection↩︎

Since PersonaJudge simulates individual human judgments by using evaluator-specific, multi-facet demonstrations, we obtain those demonstrations through a two-stage annotation workflow designed to preserve natural judgments while capturing complementary explanatory signals. In Stage 1, evaluators complete preference tasks naturally while GUI interactions are logged automatically. In Stage 2, evaluators review interaction replays and provide think-aloud commentary about their reasoning. Full data collection details are provided in Appendix 7.

4.2.0.1 Task Interface

We implement a browser-based interface with conversational contexts and paired responses. Evaluators select among {Prefer A, Neutral, Prefer B} with neutrality as an equally valid choice. Progressive disclosure reveals information as evaluators complete milestones, with all interactions automatically timestamped using a structured event taxonomy (See 7.1).

4.2.0.2 Data Post-Processing

After multi-facet data collection, we transform raw interface interaction logs and retrospective reasoning transcripts into the structured demonstration format used by PersonaJudge. For interface telemetry, we convert the collected interaction logs into a standardized event-based representation. Each interaction is serialized as a typed event with a fixed set of fields, creating a compact but expressive description of evaluator behavior. The full post-processed telemetry scheme used in ICL is provided in 4. This representation preserves sequential interaction information while maintaining structural consistency across demonstrations. For retrospective reasoning, we organize each think-aloud script into an ordered list of section-specific reasoning entries, where each entry contains a reasoning section and its associated content (6).

4.3 Experimental Factors↩︎

We employ a \(4\times4\times4\) factorial design in which we generate simulations for three controlled factors:

• Demonstration Type: We produce simulations using the four prompt-based configurations made possible by PersonaJudge: (1) J (judgment-only); (2) J+IT (judgments with interface telemetry); (3) J+RR (judgments with retrospective reasoning); and (4) J+IT+RR (judgments with interface telemetry and retrospective reasoning data).

• Demonstration Count: We conduct simulations using a predetermined number of randomly-sampled demonstrations: (1) one demonstration, (2) two demonstrations, (3) four demonstrations, or (4) eight demonstrations.

• Simulation Model: We conduct simulations using four performant LLMs: (1) Claude-3.5-Sonnet-V2, (2) Claude-3.7-Sonnet-V1, (3) DeepSeek-R1, and (4) Amazon Nova Premier.

4.4 Experimental Setup↩︎

For each evaluator within a task, we used a disjoint split of 100 annotated instances: a 40-item demonstration pool and a 60-item validation set. All reported simulation results are computed on the validation set, while demonstrations are drawn only from the demonstration pool. The evaluation items are never reused as in-context demonstrations.

For each human judgment in the validation set, we generated 64 simulated judgments by crossing three experimental factors mentioned in 4.3. Each simulated judgment follows the two-round cascade (Algorithm 8, Appendix 11): a preference-detection round is run first, and a direction round is run only when the first round predicts a preference. For each condition, demonstrations for both rounds are sampled from the same target evaluator’s 40-item demonstration pool. Round 1 samples \(k\) demonstrations balanced between Neutral and directional preference labels, while Round 2 samples \(k\) demonstrations balanced between Prefer A and Prefer B. Exact balancing is not possible in the 1-shot setting. Two helpfulness evaluators with only a few neutral labels are handled as a direction-only special case (Appendix 10). We used a single random draw for each evaluator-condition pair and did not repeat the condition across multiple random resamples, so some variance due to demonstration composition may remain unmeasured.

For each dataset, we evaluate 21 evaluators, each with 60 validation instances across all simulated settings. This results in 80,640 simulations per dataset and 161,280 simulations in total.

4.5 Metrics and Baselines↩︎

We report three-class accuracy over {Prefer A, Neutral, Prefer B} by comparing each simulated judgment—composed from the two-round cascade—with the corresponding evaluator’s original human judgment on the same validation instance. This metric assesses how accurately the model reproduces an individual evaluator’s ground-truth decisions, rather than agreement with an aggregated or consensus label. We preserve the full judgment space instead of collapsing evaluations into binary preferences as in prior works [5], [7].

4.5.0.1 Primary baselines

We use Random Selection, which chooses uniformly among the three labels (expected accuracy \(1/3\)). The Base Judge (Model Judgment Baseline) is a zero-shot LLM-as-Judge that produces a judgment with no evaluator-specific demonstrations and no persona, using the same two-round protocol. This baseline isolates the model’s prior, against which the contribution of evaluator-specific demonstrations is measured.

4.5.0.2 Controls and reference predictors

To interpret how PersonaJudge captures individual variation (5.1.3), we use one control and three reference predictors, again scored against each evaluator’s own labels. (i) The cross-evaluator control holds the model, demonstration type, and shot count fixed but draws demonstrations from non-target evaluators, averaging over all available non-target evaluators of each item, isolating the benefit of evaluator-matched demonstrations from that of demonstrations in general. (ii) The oracle consensus baseline predicts, for every evaluator, an item’s majority label among its three human annotations; it is an “oracle” because it uses held-out human labels the model never observes, and serves as the strongest possible group prior. (iii) The global majority-class baseline always predicts the single most frequent label across the entire task, pooled over all evaluators (e.g., Neutral on Harmlessness). (iv) The per-evaluator majority-class baseline predicts each individual’s own most frequent label—the strongest label-frequency predictor available per person.

5 Results and Analysis↩︎

Our empirical study addresses three questions:

• [RQ1]: Can evaluator-specific demonstrations improve over the Base Judge when simulating a specific evaluator?

• [RQ2]: How do complementary evaluator-specific signals affect simulation performance, and how do these effects vary with model choice and demonstration count?

• [RQ3]: How does simulation fidelity vary across evaluators, and which evaluator-level properties explain that variation and transfer across tasks?

5.1 Main Results↩︎

5.1.1 RQ1: Overall Effectiveness↩︎

2 presents our core results. We compare Random Selection, the Base Judge, and PersonaJudge, which conditions on evaluator-specific demonstrations. PersonaJudge results are averaged over the 64 simulation configurations defined by 4 demonstration types, 4 demonstration counts, and 4 models for each dataset.

The Base Judge outperforms random selection (\(0.333\)), reaching \(0.452\) on Harmlessness and \(0.496\) on Helpfulness on average. PersonaJudge improves on these non-personalized judges, reaching \(0.480\) on Harmlessness and \(0.510\) on Helpfulness (\(+2.8\) and \(+1.4\) points on average).

These averages are deliberately conservative: they pool all 64 model \(\times\) demonstration-type \(\times\) shot combinations, including suboptimal combinations. Comparing each combination against its own model’s Base Judge, \(43\) of \(64\) combinations improve on Harmlessness and \(50\) of \(64\) on Helpfulness, and the J+RR family improves in \(14/16\) and \(16/16\) combinations, respectively. The combinations that fail to beat their Base Judge are concentrated in the IT conditions (\(16\) of the \(21\) Harmlessness misses and \(9\) of the \(14\) Helpfulness misses involve telemetry), foreshadowing the negative telemetry effect in 5.1.2.

To verify that these gains reflect personalization rather than demonstrations in general, we compare PersonaJudge against a cross-evaluator control that uses demonstrations from a non-target evaluator while holding the demonstration type, model, and shot count fixed. PersonaJudge outperforms this control by \(+0.028\) on Harmlessness (\(0.477\) vs.\(0.450\), \(p=0.019\)) and \(+0.044\) on Helpfulness (\(0.515\) vs.\(0.471\), \(p<0.001\)).¹ The advantage is positive for most evaluators and becomes strongest with 4–8 demonstrations, suggesting that evaluator-specific examples provide genuine personalization signal rather than merely additional labeled context.

5.1.2 RQ2: Signal and Factor Patterns↩︎

1 reports marginal average simulation accuracy for each experimental factor. Each block isolates one factor while averaging over the other two: Demonstration Type averages over all models and counts, Model averages over all demonstration types and counts, and Demonstration Count averages over all models and demonstration types.

5.1.2.1 Evaluator-specific signal effects

1 shows consistent demonstration-type patterns across both datasets: retrospective reasoning is the most useful complementary signal, and interface telemetry the least. J+RR achieves the highest average accuracy on both tasks (\(0.505\) Harmlessness, \(0.537\) Helpfulness), whereas J+IT is worst and underperforms judgments alone (\(0.457\), \(0.492\)); even combined with reasoning, telemetry pulls J+IT+RR below J+RR on both tasks. The demonstration-type effect is statistically robust (Friedman \(\chi^2(3)=22.4\)/\(20.0\), \(p<0.001\); J+RR \(>\) J+IT under Wilcoxon, \(p\le0.012\), \(d\ge0.85\); 14).

5.1.2.2 Model and demonstration-count effects

Both factors matter less than demonstration type. Model choice is weak and task-dependent: the only significant pair is Claude-3.5-Sonnet \(>\) Claude-3.7-Sonnet on Harmlessness (\(p=0.002\)), with no significant model differences on Helpfulness. Accuracy rises with demonstration count but plateaus after 4 shots (Friedman \(p<0.05\) on both tasks; 1-shot significantly below 4- and 8-shot), so stable simulation is achievable with relatively few demonstrations.

5.1.2.3 Best overall configuration

2 highlights the strongest condition combinations across tasks. The top settings concentrate in Claude-family models and all use retrospective reasoning (J+RR), reinforcing the factor-level finding. We recommend a single configuration, Claude-3.5-Sonnet with 8-shot J+RR, which reaches \(0.581\) on Harmlessness (\(+9.9\) points over its Base Judge) and \(0.558\) on Helpfulness (\(+5.8\) points). This configuration significantly improves over its Base Judge on both tasks (Wilcoxon \(p=0.046\) and \(p=0.008\)); the smaller Helpfulness gain is more significant because it is more uniform across evaluators, whereas the larger Harmlessness gain is more variable. Claude-3.7-Sonnet with 2-shot J+RR attains marginally higher Helpfulness accuracy (\(0.568\)), but its gain over the Base Judge is not significant (\(p=0.057\)).

5.1.3 RQ3: Evaluator-Level Simulatability↩︎

To understand why some evaluators are easier to simulate than others, we analyze per-evaluator accuracy and its relationship to task performance and judgment behavior.

5.1.3.1 Variation in per-evaluator fidelity

Individual simulation accuracy varies widely. Averaged over all configurations, per-evaluator accuracy ranges from \(0.375\) to \(0.565\) on Harmlessness and from \(0.386\) to \(0.655\) on Helpfulness, indicating substantial evaluator-level variation in simulation fidelity.

5.1.3.2 Genuine but modest individual capture

Does this variation reflect genuine individual simulation, or just a better group answer? The decisive test is deviation items—where an annotator disagrees with the item’s consensus label (\(20.2\%\) / \(23.3\%\) of items), so any group- or consensus-based predictor scores \(0\) by construction. On these, PersonaJudge still recovers the annotator’s exact label \(0.367\) / \(0.360\) of the time—accuracy unattainable from group-level information—and its deviations are individual-aligned in both incidence and direction (Appendix 15). The captured signal is nonetheless modest: PersonaJudge exceeds a global majority-class baseline but not a per-evaluator one (each individual’s own most frequent label; \(\Delta={-}0.019\), \(p=0.95\) Harmlessness; \(\Delta={+}0.042\), \(p=0.14\) Helpfulness), so it complements rather than replaces simpler per-person predictors.

5.1.3.3 Judgment style predicts difficulty

Two facets of an evaluator’s judgment style explain who is hard to simulate. First, neutral usage: Neutral is modal in Harmlessness (\(43.9\%\)) but rarer in Helpfulness (\(31.1\%\), vs.\(34.0\%\) Prefer A / \(34.9\%\) Prefer B), and a higher neutral response rate is associated with lower accuracy—strongly for Helpfulness (\(r={-}0.894\)) but only weakly for Harmlessness (\(r={-}0.269\)). Second, divergence from consensus: an evaluator’s deviation rate—the share of items where their label differs from the majority of the other annotators²—is a significant negative predictor of accuracy on both tasks (\(r={-}0.463\), \(p=0.034\) Harmlessness; \(r={-}0.676\), \(p<0.001\) Helpfulness), and the negative association persists after partialling out neutral usage (Harmlessness \(r={-}0.576\), \(p<0.01\); Helpfulness \(r={-}0.427\), \(p=0.06\)). Idiosyncratic disagreement therefore adds difficulty beyond neutrality: evaluators with clear, consensus-aligned preferences are easier to simulate than those who abstain or diverge.

5.1.3.4 A stable trait with a task-dependent effect

For the 10 evaluators who completed both tasks, simulation fidelity itself does not transfer across tasks (\(r=0.181\), \(p=0.616\)). What transfers is the evaluator’s neutral-usage tendency: their Neutral-selection rate is strongly correlated across the two tasks (\(r=0.728\), \(p<0.05\); 3). Neutral usage is a stable, trait-like property of an evaluator—but because its effect on difficulty is task-dependent (strong for Helpfulness, weak for Harmlessness, above), this stable trait does not make simulatability itself stable across tasks. We interpret this cautiously given the limited overlap set (\(n=10\)).

5.2 Discussion↩︎

5.2.0.1 Why does retrospective reasoning help more than interface telemetry?

Retrospective reasoning likely helps because it externalizes the internal logic of a decision—how evaluators interpret context, weigh trade-offs, and justify uncertainty—in a text form LLMs can readily use, aligning the model with not only what was judged but why. Interface telemetry, by contrast, captures procedural behavior (clicks, reveals, dwell time) whose link to evaluative intent is indirect and varies across evaluators and tasks, making it a noisier conditioning signal. We therefore interpret this result as specific to our event-level telemetry encoding; higher-level behavioral summaries, such as dwell-time aggregates or re-read counts, may provide different or stronger signals.

5.2.0.2 Uneven but systematic simulatability

Because simulation difficulty tracks evaluator judgment style rather than noise (5.1.3), it is unlikely to close with more of the same signals: current methods reproduce decisive, consensus-aligned styles more readily than uncertain or idiosyncratic ones, and the individual signal they capture, though genuine, does not yet exceed a person’s own label tendency. Better representing ambiguity and idiosyncratic disagreement is therefore a key direction for future work.

5.2.0.3 Implications for evaluator-specific simulation

These findings suggest that evaluator-specific simulation can complement consensus toward modeling meaningful individual variation. Disagreement, neutrality, and diversity should not be treated as noise, but as part of the target behavior that simulation methods must capture.

Practically, complementary signals show a clear cost–benefit asymmetry: retrospective reasoning is more costly to collect than interface telemetry—roughly five times more time per item—but contributes substantially more to simulation fidelity and interpretability (per-stage annotation times in 8). Reasoning collection is therefore justified where fidelity matters most (e.g., safety or fairness auditing), whereas raw telemetry alone is unlikely to suffice for routine large-scale pipelines; how to scale per-evaluator reasoning collection to large annotator pools remains an open challenge.

6 Conclusion and Future Work↩︎

We introduce PersonaJudge, an evaluator-specific simulation framework. PersonaJudge models evaluator-specific judgments by combining categorical preferences, retrospective reasoning, and interface telemetry through in-context learning. Complementing the Base Judge, our results show that individual-level simulation is both feasible and meaningful: evaluator-specific demonstrations improve over non-personalized judges, retrospective reasoning provides the strongest benefit, and interface telemetry can even reduce performance despite its lower collection cost. We also find that simulation difficulty is systematic—driven by evaluators’ neutral usage and divergence from consensus—with neutral usage a stable cross-task trait; the individual signal captured is genuine but modest, complementing simpler per-person predictors. These findings suggest that the next step is not simply collecting more signals, but improving how we capture evaluator intent: extracting more value from low-cost telemetry and collecting richer reasoning with less user burden. More broadly, PersonaJudge illustrates a shift in LLM-as-Judge from producing pooled labels to simulating how individuals evaluate, supporting systems that are both scalable and faithful to the inherent diversity in human judgment. Beyond evaluation, such individually grounded simulations could provide fine-grained per-evaluator preference signals for reward modeling under heterogeneous human preferences [31], [32], complementing synthetic-persona datasets constructed from aggregate demographic distributions [55]–[58]. Future work can extend this line by broadening beyond pairwise HH-style judgments, scaling to more diverse evaluator populations, and integrating adaptive memory or fine-tuning strategies tailored to richer behavioral traces.

Limitations↩︎

In this work, we focus on pairwise preference judgments over conversational follow-up responses in the Helpful and Harmless setting. This scope provides a controlled and practically important testbed for evaluator-specific simulation, but we have not yet tested generalization to other evaluation formats, domains, or modalities. Further, our dataset includes 32 trained annotators and 4,200 total judgments. This is substantial for a process-rich collection protocol, but broader population coverage remains a next step; expanding evaluator diversity was beyond the bandwidth of this study because each instance required collecting both outcome labels and richer auxiliary traces. PersonaJudge uses in-context learning with sampled demonstrations as a strong and transparent baseline. Alternative approaches—including fine-tuning, retrieval-based memory, or sequence-aware architectures for interaction traces—are promising directions that we leave for future work due to scope constraints. Our study also omits two measurements that would benefit future work: a delayed test–retest of evaluators, which would give an empirical ceiling on simulation accuracy, and a formal workload survey (e.g., NASA-TLX) to quantify annotator fatigue. Retrospective reasoning, although cued by replaying each evaluator’s own interaction rather than recalled from memory alone, remains a post-hoc account and may partially rationalize rather than transcribe the original decision process. Finally, neutral judgments likely reflect multiple underlying decision processes that are not explicitly disentangled in the current framework. Developing ambiguity-aware modeling for non-commitment and balanced evaluation is therefore an actionable next direction.

Ethical Considerations↩︎

Our prompt is released for reproducibility, but PersonaJudge may be potentially misused when treated as a replacement for real human evaluators. Because uncertainty and borderline cases remain challenging for PersonaJudge to simulate reliably, overreliance on PersonaJudge could misrepresent edge cases and bias downstream decisions. Further, the collection and use of reasoning traces and interface telemetry introduce privacy considerations. These limitations indicate that any practical deployment should be accompanied by transparency, consent, and human oversight.

7 Data Collection↩︎

7.1 User Interface Implementation Details↩︎

We implement PersonaJudge data collection system as a managed AWS service. The system includes a lightweight browser-based front end that standardizes task presentation and logs interface telemetry, and a back end that stores multi-facet evaluation traces—including categorical judgments, interaction logs, and retrospective reasoning data—in a structured format for downstream analysis.

7.1.0.1 Stage 1 Interface (4)

The Stage 1 interface is designed for preference evaluation. It presents the conversation context in the left panel, two candidate AI-generated responses in the upper-right panel, and a three-point judgment scale in the lower-right panel for selecting the more helpful/harmless response or indicating a tie. Evaluators complete the task in this interface while interaction traces are logged automatically.

7.1.0.2 Stage 2 Interface (5)

The Stage 2 interface is designed for retrospective think-aloud collection. It presents a replay of the evaluator’s earlier interaction trace in the main panel, together with playback controls for reviewing the session. A side panel contains structured input sections for explaining conversation understanding, response evaluation, and final judgment selection, enabling evaluators to provide reasoning while revisiting their prior actions.

7.2 Dataset Selection and Assignment↩︎

For our study, we randomly selected 700 unique (i.e., distinct dialogue threads without shared conversation history) conversations from each dataset, limiting conversation length to 3-5 turns. This approach ensured that participants could adequately comprehend the conversational context while preventing excessive cognitive load. All responses are randomly shuffled to prevent order bias in participant evaluations. We implemented additional quality control measures by filtering out empty responses and ensuring that at least 60% of words in each response appeared in the Google common English word list.³

For each dataset, all 700 conversations were annotated by three different evaluators to improve label reliability. To manage the workflow efficiently, we partitioned the 700 conversations into 70 batches of 10 conversations each. These 70 batches were randomly and evenly assigned to 21 evaluators, with each batch completed independently by three evaluators. To reduce potential order effects, the presentation order of conversations within each batch was randomly shuffled before evaluators began the task.

7.3 User Study Procedure↩︎

The user study procedure consisted of onboarding and tutorial training, followed by a two-stage annotation workflow. Our 32 evaluators were professional data annotators from an in-house data annotation team at our institution, with prior annotation experience; they were not anonymous crowdsourcing workers. Evaluators accessed the system through a web link and logged in using their assigned credentials. All judgments were completed remotely on the evaluators’ own laptops.

7.3.0.1 Onboarding

Evaluators were first introduced to the study goals, task procedures, and data handling practices. Informed consent was obtained prior to participation. Evaluators were told that participation was voluntary, that they could withdraw at any time without penalty, and that their audio recordings would be stored confidentially.

7.3.0.2 Tutorial and training

Evaluators were then introduced to the system workflow through either a live Zoom onboarding session or a recorded tutorial video, depending on when they joined the study. After this introduction, they were presented with 10 tutorial items designed to familiarize them with the interface, task format, and retrospective reasoning procedure. These tutorial items were reviewed to ensure that evaluators understood the workflow before beginning the main annotation tasks.

7.3.0.3 Main annotation tasks

Each evaluator completed 100 assigned evaluation instances, organized into 10 batches. Because providing retrospective reasoning imposed substantial mental effort, evaluators were allowed to pause after any batch and resume later at their convenience. They were also encouraged to ask questions at any point during the annotation process.

8 Annotation Time↩︎

We quantify the per-item annotation effort of each collection stage from the interaction logs (\(n=2{,}100\) tasks per dataset; 21 evaluators \(\times\) 100 items). Stage 1 time (categorical judgment with interface telemetry, collected together) is measured from the interface telemetry as hands-on time—from the start of an item to its final rating_selected event—with idle pauses longer than \(30\) s removed so the figure reflects active engagement rather than breaks. Stage 2 time (retrospective think-aloud) is the duration of each evaluator’s reasoning sections (conversation understanding, response evaluation, and judgment).

3 reports the figures. Collecting a categorical judgment takes under a minute per item (\(\sim\)​56–57 s), and interface telemetry adds no marginal cost because it is logged passively during Stage 1. In contrast, retrospective reasoning is roughly five times more expensive to collect (\(\sim\)​4.6–5.1 min per item in Stage 2), bringing the total to about \(6\) min per item. This cost asymmetry underlies the practical trade-off discussed in §5.2.0.3: the most informative signal (retrospective reasoning) is also by far the most expensive to collect, whereas the cheapest signal (telemetry) contributes least to simulation fidelity.

9 Demonstration Format↩︎

9.0.0.1 Processed Interface Telemetry

4 presents the structured interface telemetry schema used in PersonaJudge demonstrations.

4pt

9.0.0.2 Formatted Retrospective Reasoning

6 presents the formatted retrospective reasoning used in the PersonaJudge demonstration.

9.0.0.3 ICL Demonstration Format

7 presents the overall structure of a PersonaJudge demonstration.

10 Simulation Prompt↩︎

We generate each simulated judgment through a two-round procedure that decomposes the three-class judgment into two binary decisions. In the first round, the model decides whether the evaluator expresses a preference at all (preference vs. no preference / neutral). In the second round, invoked only when the first round indicates a preference, the model decides the direction (prefer Option A vs. prefer Option B). The two outputs are composed into the final three-class label {Prefer A, Neutral, Prefer B}: a first-round “no preference” yields Neutral, otherwise the second-round direction determines Prefer A or Prefer B.

Demonstrations are converted to match each round. For the first-round prompt, each demonstration’s label is mapped to a binary preference indicator: a neutral judgment becomes “no preference” (1) and any directional judgment (prefer A or prefer B) becomes “has a preference” (2). For the second-round prompt, only demonstrations in which the evaluator expressed a preference are retained, each labeled with its direction: prefer Option A (1) or prefer Option B (2).

10.0.0.1 Direction-only special case.

For two helpfulness evaluators whose labels contain a few Neutral instances (evaluators 1 and 9), the preference-detection step is degenerate. For these evaluators, simulation proceeds directly with the direction round, and the composed labels are therefore restricted to Prefer A or Prefer B.

We use a single shared template for both rounds and both tasks; only the Judgment Scale block changes with the round and task, as shown below. Each instantiated prompt includes only the scale block for its task and round; the alternatives are shown together here for compactness.

You are a simulation agent specifically trained to replicate the judgment preferences in a preference judgment task involving model-generated responses. Your core function is to analyze pairs of responses and provide a judgment that aligns to the individual's provided demonstrations, rather than making a judgment based on your own biases, preferences, and tendencies.
Your task is to:
1. Carefully study the example judgments provided in the <examples> tags
2. Identify the rating patterns and preference criteria shown in these examples
3. Apply the same judgment framework to rate new response pairs
4. Output your rating in a way that maintains consistency with the example patterns

Judgment Scale:
[Round 1 -- Preference Detection]
[For Helpfulness Dataset]
1 = No preference (Options A and B are equally helpful)
2 = Has a preference on which option is more helpful
[For Harmlessness Dataset]
1 = No preference (Options A and B are equally harmless)
2 = Has a preference on which option is more harmless
[Round 2 -- Preference Direction; used only when Round 1 = has a preference]
[For Helpfulness Dataset]
1 = Prefer Option A as more helpful
2 = Prefer Option B as more helpful
[For Harmlessness Dataset]
1 = Prefer Option A as more harmless
2 = Prefer Option B as more harmless

Important: Your role is to simulate an individual's judgment patterns based on the provided demonstrations. You should not make independent assessments based on your own criteria. Your success is measured by how well your ratings match the patterns in the provided demonstrations.

Individual's Examples:
<examples>
    <demonstration_1>
    ...
</examples>

Current Task:
<current_task>

Provide your simulated judgment in the format:
<judgment>X</judgment>
where X is the rating number (1 or 2).

11 Simulation Procedure↩︎

Algorithm 8 summarizes the PersonaJudge two-round simulation for one query; the study runs it for every validation item under each model \(\times\) demonstration-type \(\times\) shot condition (§4.4). Each demonstration carries the evaluator’s stored label and, depending on the condition, its interface telemetry and/or retrospective reasoning.

12 Simulation Detailed Evaluator-Level Results↩︎

[tab:evaluation_results_harmless,tab:evaluation_results_helpful] report evaluator-level simulation accuracy for the harmlessness and helpfulness datasets, respectively. All accuracies are computed by comparing simulated judgments against the corresponding ground-truth judgments of the same evaluator on the validation set.

The Model Judgment Baseline columns report zero-shot LLM-as-Judge baselines (no demonstrations). These prompts include only the task instruction and the query instance. All remaining columns report PersonaJudgeresults. Because listing all 64 evaluator-specific conditions would be unwieldy, we summarize them using marginal averages over the other experimental factors. Specifically, under Model, each value is averaged over all demonstration types and demonstration counts for that model; under Demonstration Type, each value is averaged over all simulation models and demonstration counts for that demonstration type; and under Demonstration Count, each value is averaged over all simulation models and demonstration types for that demonstration count. The Avg row reports the mean across evaluators.

13 Base Judges and PersonaJudge Preference Distribution↩︎

9 shows the individual evaluator preference distribution across Prefer A, Neutral, and Prefer B for both harmlessness and helpfulness datasets.

[tab:table_distribution_single_turn_harmless,tab:table_distribution_single_turn_helpful] show the preference distribution rate for the Base Judge, PersonaJudge, and human on both harmlessness and helpfulness datasets.

14 Significance Tests and Controls↩︎

We treat the evaluator as the unit of analysis (\(n=21\) per task): each evaluator contributes one accuracy per condition, and all condition comparisons are paired across evaluators, matching the across-evaluator standard deviations reported in the main text. We report non-parametric tests as primary (Wilcoxon signed-rank for paired contrasts; Friedman for omnibus) and corroborate with parametric analogues (one-sample / paired \(t\); repeated-measures ANOVA). Pairwise families are Holm-corrected, and the 64-condition family is additionally controlled with Benjamini–Hochberg FDR (\(\alpha=0.05\)). Composed three-class judgments are coded as \(\{1{=}\text{prefer A},\,2{=}\text{no preference},\,3{=}\text{prefer B}\}\) throughout the analysis.

14.1 Alignment↩︎

Because the Base Judge already far exceeds the chance rate (\(1/3\)), the informative bar for alignment is each evaluator’s own label tendency, not chance. Per-evaluator simulatability is not significantly above a per-evaluator majority-class baseline that always predicts each individual’s own most frequent label (9); it does, as a sanity check, clear the chance rate in both tasks and at every factor level (one-sample \(t\) and Wilcoxon, all \(p<0.001\)). PersonaJudge therefore aligns with individuals well beyond chance, but we do not find evidence that it improves over each individual’s marginal label tendency. This supports positioning PersonaJudge as a complement to, rather than a replacement for, simple baseline predictors.

14.2 Factor Effects: Omnibus Tests↩︎

Configuration matters overall: a Friedman omnibus across all 64 model \(\times\) shot \(\times\) demonstration-type conditions is highly significant in both tasks (\(\chi^2(63)=191.4\) Harmlessness, \(164.0\) Helpfulness, \(p<0.001\)). 10 reports per-factor effects under repeated-measures ANOVA and Friedman. Demonstration type is the most robust effect (significant under both tests in both tasks); shot count is significant in both; model choice is weak and test-dependent. The model \(\times\) demonstration-type interaction is significant in both tasks (ANOVA \(p<0.001\)), indicating the best model depends on the demonstration format; no other interaction is significant (\(p>0.16\)).

14.3 Pairwise Contrasts↩︎

11 reports the Holm-corrected Wilcoxon contrasts within each factor (Cohen’s \(d\) paired). For demonstration type, J+RR significantly exceeds both J+IT and J+IT+RR in both tasks, and J exceeds J+IT on Harmlessness. For shot count, 1-shot is significantly below 4- and 8-shot, with no significant difference between 4- and 8-shot (a plateau). For model, the only significant pair is Claude-3.5-Sonnet \(>\) Claude-3.7-Sonnet on Harmlessness.

14.4 Per-Configuration Gains over the Base Judge↩︎

Beyond the directional breadth reported in §5.1.1 (43/64 Harmlessness and 50/64 Helpfulness configurations improve over their same-model Base Judge), we examine which gains survive multiple-comparison correction. Individual per-configuration gains are modest relative to the high cross-evaluator variance: after FDR correction, only 3 (Harmlessness) and 0 (Helpfulness) of the 64 configurations remain significant. The recommended configuration (Claude-3.5-Sonnet, 8-shot, J+RR) is significant as a single planned comparison (\(+0.099\), \(p=0.046\) Harmlessness; \(+0.058\), \(p=0.008\) Helpfulness). Consistent with the telemetry finding, one telemetry-bearing configuration is significantly worse than its Base Judge (Claude-3.7-Sonnet, 1-shot, J+IT+RR; \(-0.068\), FDR-significant). Across all \(2{,}016\) pairwise comparisons among the 64 conditions, no single configuration is significantly best after correction (Holm \(0/2016\) on both tasks; FDR \(117/2016\) Harmlessness and \(0/2016\) Helpfulness).

14.5 Cross-Evaluator Personalization Control↩︎

To isolate the personalized component from the general benefit of demonstrations, we compare PersonaJudge against a control that holds the model, demonstration type, and shot count fixed but draws the demonstrations from non-target evaluators. Because each item is annotated by several evaluators and PersonaJudge already produced a prediction for each, the control reuses those existing predictions (no extra inference): for target \(e_i\) and item \(x\), we score each non-target evaluator \(e_j\)’s prediction (same model/type/shot) against \(e_i\)’s own gold and average over all such evaluators. It is leakage-free, as PersonaJudge already excludes the test item from each evaluator’s demonstrations, and is computed on the matched multi-annotator subset (\(84.1\%\) of pairs).

PersonaJudge outperforms the control overall by \(+0.028\) on Harmlessness (\(0.477\) vs. \(0.450\), \(p=0.019\)) and \(+0.044\) on Helpfulness (\(0.515\) vs.\(0.471\), \(p<0.001\)). The advantage is negligible at one demonstration and grows with shot count (12)—the signature of genuine personalization rather than of additional labeled examples—is positive across all four judge models and all demonstration types, and is positive for 15/21 (Harmlessness) and 20/21 (Helpfulness) evaluators.

15 Error Analysis↩︎

This appendix expands the RQ3 finding (§5.1.3) that PersonaJudge captures genuine—but modest—individual variation rather than a group prior. For each item we define its consensus label as the majority of the three human annotations (items with no majority, i.e.-1-1, are excluded); an (annotator, item) pair is a deviation when the annotator’s own label differs from that consensus. Unless noted, metrics pool over all 4 models, 4 demonstration types, and 4 shot counts and are scored against each evaluator’s own gold labels.

15.1 Baseline comparison↩︎

13 compares PersonaJudge against the reference predictors of §4.5, all scored against each evaluator’s own gold labels on the consensus-eligible subset (so only the prediction rule differs across rows). PersonaJudge sits far below the oracle consensus baseline (it is not reproducing the group answer), above the global majority-class baseline, and approximately at each evaluator’s own majority-class baseline—genuine individual signal that does not yet exceed a person’s marginal label tendency.

15.2 Capturing individual deviation↩︎

The empirical human deviation rate—the share of (annotator, item) pairs where the annotator disagrees with the item’s consensus label—is \(20.2\%\) (Harmlessness) and \(23.3\%\) (Helpfulness). Deviation items are the decisive test of individual capture: because the annotator’s gold differs from the consensus, any group- or consensus-based predictor scores \(0\) on them by definition, so any accuracy here must come from individual-level signal. We quantify capture with three complementary metrics (14).

Deviation accuracy is the fraction of deviation items on which PersonaJudge predicts the annotator’s exact label. At \(0.367\) / \(0.360\), PersonaJudge gets right more than a third of the cases a group prior cannot.

Alignment lift asks whether PersonaJudge deviates when the annotator does. It is \(P(\text{model deviates}\mid\text{annotator deviates}) - P(\text{model deviates}\mid\text{annotator conforms})\): how much more often the model breaks from consensus on items where the individual also breaks from it, versus items where the individual agrees with the group. A positive value (\(+0.117\) / \(+0.147\)) means the model’s deviations are selective—triggered on the same items the individual deviates on—rather than random noise.

Captured direction asks whether, when PersonaJudge does deviate on a deviation item, it picks the annotator’s actual non-consensus label rather than the other one. With three labels, ruling out the consensus leaves two options, so a random choice scores \(0.500\); PersonaJudge reaches \(0.617\) / \(0.632\), so it recovers not just that an individual deviates but the direction of the deviation.

These results show PersonaJudge’s departures from consensus are individual-aligned in both incidence (alignment lift) and direction (captured direction), and that it recovers individual labels (deviation accuracy) unattainable from group-level information. The pattern holds across all four judge models, with Nova Premier marginally the weakest.

15.3 Confusion matrices and per-class metrics↩︎

15 reports row-normalized confusion matrices (diagonal = recall). The Neutral (no-preference) class is by far the hardest to recover (recall \(0.356\) / \(0.242\)) and is most often misread as a directional preference; direct A\(\leftrightarrow\)B reversals are comparatively rare. 16 gives per-class precision/recall/F1.

References↩︎