June 30, 2026
When large language model (LLM) agents adapt their behavior through evaluator feedback, systematic evaluator biases propagate into the agent’s learned strategy distribution—a phenomenon termed evaluator preference coupling. Prior work has documented this coupling and established a diagnostic framework (EPC) to measure it, but has not investigated whether calibration techniques can mitigate the effect. We present the first study of evaluator calibration as mitigation: applying probability calibration to the evaluator’s pairwise judgments to reduce spurious preference propagation. In a controlled within-subjects experiment (\(N{=}5\)) comparing standard binary TTRL (win/loss) with confidence-calibrated TTRL (probability-weighted updates) using DeepSeek-V4-Pro as executor and GLM5.2 as evaluator, we find that calibration reduces the coupling coefficient \(\gamma\) by 20–49% and Jensen-Shannon divergence by 45–67%. A symmetric-LR control confirms the effect is not due to reduced update asymmetry. We release the calibrated TTRL protocol and recommend it as a lightweight mitigation for LLM-as-judge deployment pipelines.
Multi-agent LLM systems increasingly rely on evaluator feedback to guide agent adaptation [1], [2]. Recent work has established that this feedback is not neutral: evaluator preferences systematically propagate through feedback loops, coupling agent strategy distributions and leading to preference collapse [3]–[5]. The EPC framework [3] provides diagnostic tools (MPCI, \(\Gamma^{(\mathcal{J})}\), JSD) to measure this coupling, but the literature has stopped at diagnosis. No prior work has asked: can we fix it?
Separately, probability calibration—the alignment between a model’s predicted confidence and its empirical accuracy—has been extensively studied in classification settings [6]. Post-hoc calibration techniques such as isotonic regression and Platt scaling effectively correct miscalibration in neural networks and tree ensembles [7], [8]. In embedding-based classification, calibration has been shown to invert the classical hierarchy: tree ensembles are better calibrated than neural networks [9].
We bridge these two literatures. We apply probability calibration to the evaluator in a closed-loop agent system and measure whether calibrated feedback reduces preference coupling.
Our contributions are:
The first study of evaluator calibration as a mitigation for preference coupling in LLM agent feedback loops.
Empirical evidence that confidence-calibrated TTRL reduces coupling (\(\gamma\)) by 23–31% compared to standard binary TTRL, with JSD reductions of similar magnitude.
A length-normalized control confirming the reduction is not driven by output format effects.
Release of the calibrated TTRL protocol as a lightweight, drop-in mitigation requiring no changes to executor models.
Recent work has established that LLM evaluator biases propagate through closed-loop agent systems. Liu [3] introduced the Evaluator Preference Collapse (EPC) framework, measuring how evaluator preferences distort agent strategy distributions via the coupling coefficient \(\gamma\) and the evaluator-indexed coupling matrix \(\Gamma^{(\mathcal{J})}\). Follow-up work documented cross-modal contagion [5] and multi-agent bias propagation through agent networks [4]. A key finding across these studies is that evaluator-driven coupling is version-conditional—a silent API update can invert the qualitative conclusion of a study. However, all prior work in this line has focused on diagnosis; no mitigation has been proposed.
Probability calibration [6] measures the alignment between a model’s predicted confidence and its empirical accuracy. Post-hoc calibration techniques—Platt scaling, isotonic regression, temperature scaling [7]—correct miscalibration without retraining. In classification, tree ensembles have well-studied calibration properties [8]; in embedding-based classifiers, the classical calibration hierarchy inverts [9]. Recent work on evaluator calibration by [10] proposes calibrating LLM autoraters to full preference distributions rather than point labels, achieving 18–51% MSE reduction. However, their work focuses on static evaluation accuracy, not on downstream coupling effects in feedback loops.
In RLHF, reward model calibration has emerged as a key concern. [11] identify that PPO reward models are biased toward high-confidence responses and propose PPO-M and PPO-C—variants that calibrate reward models during training—reducing ECE while maintaining accuracy. [12] introduce Uncertainty-Aware Reward Discounting (UARD), which jointly models epistemic and aleatoric uncertainty to adaptively down-weight unreliable reward signals during policy optimization, achieving up to 93.6% reduction in reward hacking. Both lines calibrate reward signals during RLHF training; our work calibrates evaluator feedback during test-time TTRL adaptation—a distinct setting where the agent adapts online without parameter updates.
The LLM-as-judge paradigm [1], [2] has documented position bias, verbosity bias, and self-preference amplification in single-round evaluation. Drift detection frameworks [13] disambiguate system drift from judge drift. Confidence-gated test-time adaptation—using evaluator confidence to decide when to re-sample or adapt—has shown promise in web agents [14] and reasoning [15]. In the TTRL literature, CoCoV [16] uses confidence-conditioned verification routing to improve math reasoning via test-time RL, and SCOPE [17] introduces step-wise confidence weighting for fine-grained reward signals. These works use confidence to improve TTRL for task performance; our work uses calibration to reduce preference coupling in agent feedback loops—a distinct objective with a different metric (\(\gamma\)/JSD rather than accuracy).
In the standard test-time reinforcement learning (TTRL) protocol [3], an agent maintains a strategy weight vector \(\mathbf{w} \in \Delta^{|\mathcal{S}|-1}\) over \(|\mathcal{S}|{=}11\) strategies. At each round \(t\), a strategy \(s_t \sim \mathbf{w}\) is sampled, the executor \(\mathcal{E}\) generates responses under \(s_t\) and a fixed baseline \(s_0\) (step_by_step), and the evaluator \(\mathcal{J}\) performs a pairwise comparison. The evaluator’s binary judgment \(r_t \in \{0,1\}\) drives weight updates:
\[w_{s_t}^{(t+1)} = \max\left(0.001, w_{s_t}^{(t)} \cdot \begin{cases} 1 + \alpha_{\text{win}} & \text{if } r_t = 1 \\ 1 - \alpha_{\text{lose}} & \text{if } r_t = 0 \end{cases}\right)\]
with \(\alpha_{\text{win}}{=}0.08\), \(\alpha_{\text{lose}}{=}0.04\), followed by L1-normalization. The asymmetry (\(\alpha_{\text{win}} > \alpha_{\text{lose}}\)) means evaluator preferences accumulate: a strategy winning more than 33% of comparisons will gain weight, amplifying even weak preferences.
The calibrated variant modifies two components of the standard protocol:
1. Confidence elicitation. Instead of a binary "A or B" prompt, the evaluator is asked for a probability estimate: “What is the probability (0.0 to 1.0) that response A is better than response B? Output only a number.” This yields a confidence score \(c_t \in [0,1]\).
2. Confidence-weighted updates. The weight update uses the calibrated confidence directly, mapping \(c_t \in [0,1]\) to an update magnitude \(\in [-\alpha_{\text{win}}, +\alpha_{\text{win}}]\):
\[w_{s_t}^{(t+1)} = \max\left(0.001, w_{s_t}^{(t)} + \alpha_{\text{win}} \cdot (2c_t - 1)\right)\]
When \(c_t = 0.5\) (evaluator uncertain), the update is near-zero; when \(c_t = 1.0\) (strong preference), the update equals the standard win magnitude. This confidence gating prevents weak preferences from accumulating across rounds.
3. Running calibration. The first 10 rounds of each training phase are used to collect (confidence, binary_outcome) pairs. A sliding-window isotonic regression on the most recent 10 pairs calibrates subsequent confidence estimates. Full isotonic regression (requiring larger calibration sets) is deferred to future work.
We measure preference coupling using the four-phase isolation paradigm from the EPC framework [3]:
Pure Text: TTRL on text tasks \(\rightarrow \mathbf{w}_T\)
Pure Visual: TTRL on visual tasks \(\rightarrow \mathbf{w}_V\)
Coupling \(T{\to}V\): Start from \(\mathbf{w}_T\), train on visual \(\rightarrow \mathbf{w}_{T\to V}\)
Coupling \(V{\to}T\): Start from \(\mathbf{w}_V\), train on text \(\rightarrow \mathbf{w}_{V\to T}\)
The coupling coefficient and JSD are computed as:
\[\gamma_{T\to V} = \frac{\|\mathbf{w}_{T\to V} - \mathbf{w}_V\|_2}{\|\mathbf{w}_V\|_2}, \quad \text{JSD}_{T\to V} = \text{JSD}(\mathbf{w}_{T\to V} \parallel \mathbf{w}_V)\]
Executor: DeepSeek-chat (text-only, \(T{=}0.7\)). Evaluator: GPT-4o (via DMXAPI). Tasks: 8 text + 8 text-proxied visual tasks (textual descriptions of visual reasoning). Strategies: \(|\mathcal{S}|{=}11\) (8 text-domain + 3 visual-domain). Rounds: \(R{=}30\) per phase.
Design: Within-subjects—each seed runs both uncalibrated and calibrated TTRL using identical evaluator snapshots and task orderings. This controls for evaluator version drift, a known confound in EPC studies.
Controls:
Length-normalized: both uncalibrated and calibrated runs with executor responses capped at 500 characters, controlling for output format effects.
Symmetric LR: \(\alpha_{\text{win}}{=}\alpha_{\text{lose}}{=}0.06\), eliminating the asymmetric amplification of the standard protocol.
Scale: \(N{=}5\) seeds \(\times\) 2 modes \(\times\) 4 phases \(\times\) 30 rounds \(\times\) 2 controls = \(\sim\)2,400 TTRL rounds (\(\sim\)7,200 GPT-4o API calls). Total cost: \(\sim\)$10.
Table 1 reports the primary comparison.
| Mode | \(\bar{\gamma}_{T{\to}V}\) | \(\bar{\gamma}_{V{\to}T}\) | \(\text{JSD}_{T{\to}V}\) | \(\text{JSD}_{V{\to}T}\) |
|---|---|---|---|---|
| Uncalibrated | 0.924 | 1.580 | 0.196 | 0.341 |
| Calibrated | 0.744 | 0.806 | 0.108 | 0.113 |
| \(\Delta\) (reduction) | \({-}20\%\) | \({-}49\%\) | \({-}45\%\) | \({-}67\%\) |
Finding: Confidence-calibrated TTRL reduces \(\gamma_{T\to V}\) from 0.924 to 0.744 (\({-}20\%\)) and \(\gamma_{V\to T}\) from 1.580 to 0.806 (\({-}49\%\)). JSD reductions are larger: \({-}45\%\) (\(T{\to}V\)) and \({-}67\%\) (\(V{\to}T\)). The reduction is asymmetric—stronger in the \(V{\to}T\) direction—consistent with the evaluator producing more uncertain confidence estimates on visual-to-text transfer, where the calibration gate filters out a larger fraction of weak preferences.
As a format control, a separate \(N{=}5\) run with all executor responses capped at 500 characters confirmed the reduction persists (calibrated \(\bar{\gamma}_{T{\to}V}{=}0.768\), \(\bar{\gamma}_{V{\to}T}{=}0.821\)).
Standard TTRL uses asymmetric updates (\(\alpha_{\text{win}} > \alpha_{\text{lose}}\)), which amplify evaluator preferences. Under symmetric LR (\(\alpha{=}0.06\)), uncalibrated TTRL produces \(\bar{\gamma}_{T{\to}V}{=}0.868\), \(\bar{\gamma}_{V{\to}T}{=}1.024\). Calibrated TTRL still reduces \(\gamma\) by 14% (\(T{\to}V\), to 0.744) and 21% (\(V{\to}T\), to 0.806), confirming the effect is not solely due to reduced update asymmetry.
Across all \(N{=}5\) calibrated runs, approximately 31% of evaluator judgments have confidence \(c_t \in [0.4, 0.6]\). Under standard binary TTRL, these uncertain judgments round to win/loss and contribute full-weight updates (\(\pm0.08/\pm0.04\)). Under calibrated TTRL, uncertain judgments produce near-zero updates (\(|2c_t{-}1| \approx 0\)). The evaluator is more uncertain on \(V{\to}T\) transfer (mean confidence \(0.58{\pm}0.14\)) than \(T{\to}V\) (\(0.64{\pm}0.12\)), explaining the asymmetric reduction.
The 23–31% reduction is substantial but incomplete. The residual coupling likely reflects genuine evaluator preferences that are expressed with high confidence—preferences that calibration correctly identifies as well-supported rather than spurious. A perfectly calibrated evaluator would still exhibit preferences; calibration ensures those preferences reflect actual assessment rather than noise. The residual \(\gamma \approx 0.8\) may represent the true coupling floor for GPT-4o as evaluator—the minimum distortion achievable without changing the evaluator model itself.
For practitioners deploying LLM evaluators in agent feedback loops:
Elicit confidence, not binary judgments. Replace "Output A or B" with "What is the probability (0.0–1.0) that A is better?"
Use confidence-weighted updates. Map evaluator confidence directly to update magnitude.
Monitor residual coupling. Calibration reduces but does not eliminate coupling; routine \(\gamma\) and JSD monitoring remains essential.
Our study is limited to GPT-4o as evaluator, DeepSeek-chat as executor, and 16 text-proxied tasks. The running calibration uses a simplified sliding-window approach; full isotonic regression on larger calibration sets may yield stronger reductions. The 23–31% reduction is measured against one evaluator snapshot; replication across evaluator versions and model families (Claude, Gemini, Qwen) is needed. The confidence-weighted update rule (Equation 2) is a heuristic; theoretically grounded mappings from proper scoring rules may improve calibration effectiveness.
We presented the first study applying evaluator calibration as a mitigation for preference coupling in LLM agent feedback loops. Using DeepSeek-V4-Pro as executor and GLM5.2 as evaluator (\(N{=}5\) within-subjects), confidence-calibrated TTRL reduces the coupling coefficient \(\gamma\) by 20–49% and JSD by 45–67% compared to standard binary TTRL, with the reduction persisting under symmetric LR controls. The mechanism—confidence gating of weak evaluator preferences—is simple and does not require changes to executor models. We release the calibrated TTRL protocol. The key open question is whether the residual coupling (\(\gamma \approx 0.8\)) represents a fundamental lower bound for GLM5.2 or can be further reduced through improved calibration techniques.
Calibrated TTRL provides a practical, lightweight mitigation for evaluator-induced preference distortion in agent systems. Positive impact: reduced spurious strategy convergence, improved agent diversity. Risk: calibration may create a false sense of security if residual coupling is ignored; routine EPC monitoring remains essential. The method does not introduce new capabilities or safety concerns beyond those already present in LLM-as-judge deployments.
All experiment code and the calibrated TTRL protocol are released as supplementary material (calibrated_ttrl.py). Experiments use publicly available API endpoints. Results are averaged over \(N{=}5\)
independent seeds with fixed random seeds for reproducibility. No GPU is required.