1 Introduction↩︎

Evaluating LLM outputs, especially subjective tasks without ground truth, remains hard. A common workaround uses LLMs-as-judges as preference proxies [1], but this imports judge-model biases, creating safety and robustness risks [2].

Self-preference bias, a model’s disproportionate preference for its own outputs, scales with model size, post-training, and performance [3], [4], and persists even when authorship is hidden. This poses risks for preference tuning, domain-specific annotation, and model routing [5]–[10]. Yet most work centers on detection rather than mitigation [4], [11], with remedies largely limited to fine-tuning or style changes [3].

We mitigate this with steering vectors—lightweight, inference-time activation edits with minimal training cost [12]. Prior work shows they effectively modulate behavior, though with imperfect precision.

Our contributions are threefold: (1) We curate an evaluation set for XSUM that separates illegitimate self-preference, legitimate self-preference, and unbiased agreement using ensemble "gold" judges from diverse model families; (2) We construct steering vectors for self-preference using Contrastive Activation Addition (CAA) and a data-efficient optimization method; and (3) We show these interventions flip up to 97% of illegitimate self-preferences and shift \(P(\text{self})\) toward the impartial-judge mean \(\mu_{\text{judge}}\), outperforming prompting and DPO baselines.

2 Methods and Experiments↩︎

2.1 Demonstrating Self-Preference Bias↩︎

We first evaluate self-preference bias using a framework that disentangles it from ground-truth quality. Consider a dataset \(X=\{x_i\}_{i=1}^{|X|}\) of source articles. For each article \(x_i\), a self-evaluating model \(J\) and a comparison model \(K\) produce summaries \(y_{J,i}\) and \(y_{K,i}\). We create a pairwise evaluation set from these summaries, \(Y_{J,K}(X)=\{(y_{J,i},\,y_{K,i})\}_{i=1}^{|X|}\). Using this set, we ask model \(J\) to determine the better summary for each item, writing \(v_i\in\{y_{J,i},\,y_{K,i}\}\). We define self-preference bias as the probability-weighted difference in selections, averaged over the dataset.

\[\mathtt{bias}(J, X) = \frac{1}{|X|} \sum_{i = 1}^{|X|} \left( P(v_i = y_{J,i}) - P(v_i = y_{K,i}) \right)\] To separate bias from genuine quality, we follow [13] and generate ground-truth labels using a set of gold judges \(G=\{G_1,\dots,G_n\}\) from different model families. For each item \(i\), the gold vote \(g_i\in\{y_{J,i},y_{K,i}\}\) is the majority preference of \(G\) between the two candidates \((y_{J,i},y_{K,i})\) in \(Y_{J,K}(X)\). We then define a judge score over \(X\) that measures objective quality differences between models \(J\) and \(K\): \(\mathtt{score}\!\big(G, Y_{J,K}(X)\big)=\tfrac{1}{|X|}\sum_{i=1}^{|X|}\mathbf{1}[\,g_i=y_{J,i}\,]\), i.e., the fraction of items where the gold judges prefer \(J\)’s summary. With gold labels, each evaluation of \(x_i\) by model \(J\) falls into one of three outcomes: illegitimate self-preference \((v_i=y_{J,i},\,g_i=y_{K,i})\), legitimate self-preference \((v_i=y_{J,i},\,g_i=y_{J,i})\), and unbiased agreement \((v_i=y_{K,i},\,g_i=y_{K,i})\). Concretely, illegitimate self-preference: \(J\) chooses its own summary while the gold judges prefer the other model’s summary; legitimate self-preference: both \(J\) and the gold judges prefer the self-evaluating model’s summary; unbiased agreement: both prefer the comparison model’s summary. To ensure alignment with human preference judgments, we manually validated all gold-judge decisions (Appendix 10).

2.1.0.1 Datasets

We evaluate on XSUM [14], a subjective summarization task with clear quality criteria. We sample 1,000 articles, generate summaries with Llama-3.1-8B-Instruct and GPT-3.5, and, using §2.1, compute the ground-truth mean \(\mu_{\text{judge}}=\texttt{score}(G,Y_{J,K}(X))\). We then plot judge \(J\)’s baseline probability of selecting its own output, confirming persistent self-preference in the aware setting where \(J\) is told which summary it wrote (Fig.1). While we focus our steering efforts on summarization, Appendix 11 shows preliminary investigations into different domains.

We evaluate both the self-preferring judge \(J\) and gold judges \(G\) by running each prompt twice with different summary orderings, discarding items that demonstrate positional bias [2]. We use two prompts: an unaware prompt that hides authorship, and an aware prompt that labels summaries as "your response" vs. "other model’s response". Full prompts are in Appendix 8.

2.1.0.2 Models

We select Llama 3.1-8B-Instruct [15] as our judge model, following empirical results from [16] demonstrating its capability for bias, and use GPT-3.5 [17] as a comparison model due to its performance matching Llama on summarization datasets. For gold labels, we use Phi-4 [18], DeepSeek V3 [19], and Claude 3.5-Sonnet [20].

2.2 Constructing a Steering Vector↩︎

We construct steering vectors via (1) contrastive activation addition (CAA; [21]), contrasting positive vs. negative activations to isolate a direction, and (2) optimization-based steering [22], which learns an additive vector by gradient descent on contrasted completions. We choose these for their strong results in self-recognition/refusal [16], [23].

2.2.1 Contrastive Activation Addition↩︎

CAA builds the steering vector by pairing positive and negative examples for the target behavior and averaging the hidden-state activation differences they induce.

Formally, given a dataset \(X\) of prompts \(p\) paired with completions \(c\) generated by model \(J\) with greedy sampling, we select prompts \(p_+\) that yield unbiased completions \(c_+\) and prompts \(p_-\) which yield biased completions \(c_-\), we then define the CAA vector \(\mathbf{v}_{\mathrm{CAA}}\) for a model layer \(L\) as follows:

\[\mathbf{v}_{\mathrm{CAA}} = \frac{1}{|X_+|} \sum_{(p_+,c_+) \in X_+} {h_L(p_+, c_+)} - \frac{1}{|X_-|} \sum_{(p_-,c_-) \in X_-}{h_L(p_-, c_-)}\]

where \(h_L(p, c)\) represents the activations in the residual stream at layer \(L\) after processing the prompt \(p\) and continuation \(c\). We collect activations at the last 10 token positions for all layers.

2.2.2 Gradient-based Activation Optimization↩︎

We use a contrastive promotion/suppression method defined by [22] to train an additive vector with a contrastive loss function. Let \(X\) be the input prompt, \(Y_+\) be the desired completion, and \(Y_-\) be the undesired completion. The probability of a model generating the sequence \(Y_+\) given \(X\) with the steering vector \(h\) applied to its activations is denoted as \(P_{\mathrm{model}}(Y \mid X; h)\). The optimization of \(h\) is framed as a minimization problem in a composite loss function with two terms: \[\mathcal{L}(X,Y;h) = - \log P_{\mathrm{model}}\!\left( Y_+ \,\middle|\, X; h \right) - \log\!\left( 1 - P_{\mathrm{model}}\!\left( Y_- \,\middle|\, X; h \right) \right)\]

This dual-objective loss aims to create a strong directional signal for the model’s activations. We optimize a vector at layers 14, 15, and 16 as those performed the best on our evaluations. See Appendix 7 for optimization hyperparameters.

2.3 Steering Evaluations↩︎

2.3.0.1 Baselines

We compare our constructed vectors to two realistic, approachable baselines for end users: (1) a prompt-based strategy reminding the judge model of self-preference bias (in Appendix 8.3) and (2) fine-tuning with Direct Preference Optimization [24] on all examples of self-preference bias, unbiased agreement, and legitimate self-preference. Details about finetuning can be found in Appendix 9.

2.3.0.2 Metrics

Steering is evaluated by: (1) effectiveness—the fraction of \(J\)’s biased votes that the steered judge \(J'\) corrects; and (2) stability—the fraction of \(J\)’s correct votes \(J'\) preserves (covering unbiased agreement and legitimate self-preference). Together, these measure bias suppression and preservation of valid judgments.

3 Results↩︎

We find that steering vectors can reliably reduce illegitimate self-preference and showcase high effectiveness (Table ¿tbl:tab:intervention95flips?). Three of the four steering vectors tested were able to successfully "flip" 97% of previously biased samples. Surprisingly, optimization-based steering performs comparably to CAA with far fewer examples—valuable given scarce labeled cases across our regimes. Also unexpectedly, context-unaware vectors outperformed their aware counterparts, yet both settings yielded successful flips. The cross-setting effectiveness suggests that self-preference has at least a partially linear representation in activation space. All in all, compared with prompting (0% flips) and DPO (49%), steering vectors are able to achieve substantial effectiveness gains. See Appendix 12.1 for steered examples.

However, the same vectors struggle with stability. CAA-constructed vectors in particular demonstrate little modulation indicated by their high flip rates in legitimate self preference and low flip rates in unbiased agreement in both unaware and aware settings. This provides evidence for self-preference being represented non-linearly or with multiple directions in activation space.

6pt

4 Related Work↩︎

Early work found LLMs systematically favor their own outputs [25], [26]. Measurement then improved: [7] used human-preference labels to separate illegitimate bias from justified choices; [11] tested verifiable tasks across scales; and [13] introduced gold labels from uninvolved models. We adopt this last framework to build reliable positive/negative cases for steering and evaluation.

Building on these refinements, [3] showed that frontier LLMs both recognize and favor their own outputs, with stronger recognition amplifying bias. Fine-tuning intensified both effects, underscoring risks when the same model serves as generator and judge.

In interpretability, [16] controlled self-recognition via contrastive steering. We extend this to self-preference with a pairwise setting where bias and true quality intertwine requiring reliable ground truth to separate illegitimate self-preference from justified choices.

5 Discussion and Future Work↩︎

As discussed in Section 3, our vectors are not robust to the legitimate self-preference case nor the unbiased agreement case. This could mean that illegitimate self-preference is linearly encoded while the other two are different directions either linearly or nonlinearly encoded in the residual stream. Future work should explore this possibility in depth and further improvements to our setup.

Another major confounding factor for CAA in particular was the ordering bias models exhibit in the pairwise setting. We attempt to account for this as reported in Section 2.1.0.1, however, this still obstructed our signal and could be a factor for our limited results in the LSP and agreement case. Future work should incorporate individual rather than pairwise evaluations for self-preference to circumvent this issue.

6 Steering Vector Plots↩︎

6.1 Illegitimate Self-Preference↩︎

6.2 Unbiased Agreement↩︎

6.3 Legitimate Self-Preference↩︎

7 Optimization Parameters↩︎

For our experiments, we froze the model weights, hooked an additive vector, and optimized over the top 20th percentile of prompts based on the highest probability that \(J\) would select its own answer, given that the gold label votes for model \(K\), maximizing the probability of target completion \(Y_+\) corresponding to model \(K\)’s response and minimizing the initial, biased completion \(Y_- = v_J\). We optimize using Adam on layer 14 of Llama-3.1-8b-instruct for 20 iterations with a learning rate of \(0.1\).

8 Prompts↩︎

8.1 Pairwise Evaluation Prompts↩︎

8.1.1 Unaware Setting↩︎

8.1.2 Aware Setting↩︎

8.2 XSUM Summary Generation Prompt↩︎

8.3 Prompt Intervention Baseline↩︎

9 DPO Setup↩︎

We fine-tune Llama 3.1-8B-Instruct with a Direct Preference Optimization (DPO) Low-Rank Adapter (LoRA) to predict gold standard labels in the unaware setting. We construct a preference dataset using the unbiased, gold judge consensus as the preferred answer, and the alternative choice (\(\{J, K\} \setminus g_i\)) as the unpreferred answer. We train with the adapter at rank 64 with a LoRA \(\alpha=64\) and a DPO \(\beta\) of 0.1.

The model is trained for 3 epochs using the 8-bit AdamW optimizer with a learning rate of \(5 \times 10^{-6}\) and no weight decay. We employ a linear learning rate scheduler with a warmup ratio of 0.1. The training process uses a per-device batch size of 2 with 4 gradient accumulation steps, resulting in an effective batch size of 8. For reproducibility, the random seed is set to 42.

10 Gold Judge Human Validation↩︎

To ensure the reliability of our gold-judge ensemble, we conducted a full-scale human validation of every annotated decision. The comparison in Figure 5 shows strong overall agreement between human evaluators and gold judges, with 80.2% agreement across the full dataset. Subgroup analyses reveal even higher alignment on legitimate self-preference (84.7%) and unbiased agreement (90.5%), indicating that our gold judges largely reflect consistent, human-aligned judgments in these settings. These results support the validity of our gold labels as a reasonable proxy for human preference.

However, the validation also exposes an important limitation: alignment drops sharply for the bias subset, where agreement drops to 43.8%. This highlights that while the gold-judge ensemble effectively captures objective agreement cases, it struggles to resolve more ambiguous cases—precisely the subset most relevant for self-preference. Overall, the human validation confirms that our gold judges are generally good at measuring output quality, though improvements in the bias subset remain necessary.

11 Apps Dataset Analysis↩︎

12 Sample Steered Responses↩︎

12.1 CAA, Aware Setting↩︎

References↩︎