1 Introduction↩︎

Supervised fine-tuning (SFT) on human-annotated data is a cornerstone of modern language model development [1]–[4]. In practice, annotation pipelines rely on multiple crowdworkers who vary widely in domain expertise, task familiarity, and attention [1], [4]. Despite this, the overwhelming majority of fine-tuning pipelines treat all annotations as equally reliable, aggregating labels via majority vote or simple averaging before training [5], [6]. This discards information about annotator identity and quality, and causes the model to absorb the errors of unreliable annotators directly into its parameters.

The problem is particularly acute in the low-budget crowdsourcing setting, where a small number of annotators with heterogeneous expertise label large quantities of data. When one annotator is highly reliable and two are near-random, a model trained naively on the combined annotations is learning from noise the majority of the time. Recent work has shown that annotation quality matters far more than quantity for SFT [7], yet existing methods for handling label noise [8]–[10] typically assume a fixed noise transition matrix and require either a clean validation set or strong assumptions about the noise structure—neither of which is available in the crowdsourcing setting. The question we address is whether the model can instead identify and down-weight unreliable annotators automatically, using only the observed labels and annotator identities, without any additional supervision on annotator quality.

We propose Reliable Expertise-Aware Language Model (REALM) Fine-Tuning, a method that jointly learns the model parameters and an expertise value for each annotator. The key idea is to model each observed label as a mixture between the model’s prediction and a uniform random guess, weighted by the annotator’s expertise. An annotator with high expertise pushes the model toward their label; one with low expertise contributes little signal. Crucially, expertise is never observed: it is recovered entirely unsupervised from the structure of the training data. This mixture-based formulation was introduced in the imitation learning literature [11], [12], where it was shown to allow unsupervised recovery of demonstrator quality in sequential decision-making. We adapt and extend this framework to the LLM fine-tuning setting, where supervision is token-level and expertise varies across tasks in a structured multi-task setting. To our knowledge, this mixture-based approach has not previously been applied to language model training.

We adapt the mixture objective to the LLM fine-tuning setting by restricting the model’s output distribution to the \(K\) answer-choice tokens of each multiple-choice task, and extend it to a multi-task setting where annotator expertise varies across tasks and is encoded in a learned matrix. We evaluate REALM on five multiple-choice question answering benchmarks [13]–[17], fine-tuning three sizes of Flan-T5 [2] under noisy annotations. Our experiments systematically vary the annotator expertise distribution, the noise type (uniform, asymmetric, and systematic), and the annotation mode (shared and exclusive), yielding a comprehensive picture of when and why expertise-aware training helps.

Our main contributions are as follows:

• We propose REALM, a mixture-based fine-tuning objective that jointly learns model parameters and per-annotator expertise vectors entirely unsupervised, requiring no additional supervision beyond annotator identity.

• REALM consistently outperforms naive noisy SFT across datasets, model sizes, noise regimes, and in the multi-task setting where annotator expertise varies across tasks, with an average accuracy improvement of \(+5.5\%\) and gains that grow with model capacity.

• REALM remains robust when the noise type violates the model’s generative assumptions, because unreliable annotators still produce systematically worse labels regardless of the noise structure.

2 Background and Related Work↩︎

2.1 Learning from Noisy and Crowdsourced Labels↩︎

Inferring ground-truth labels from multiple noisy annotators is a well-studied problem in machine learning. [5] introduced a seminal EM-based framework for estimating per-annotator error rates without access to gold labels, modeling each annotator through an individual confusion matrix. [6] extended this to a joint learning setting in which the classifier and annotator parameters are estimated simultaneously, showing consistent improvements over majority voting. [18] further incorporate item difficulty alongside annotator expertise, showing that both factors jointly determine label quality. These methods, however, are designed for fixed classification models and do not apply to the end-to-end fine-tuning of LLMs, where the learned representation and the annotator reliability must be optimised jointly through gradient descent.

In NLP, annotator disagreement has received growing attention, particularly for subjective tasks such as toxicity detection and sentiment analysis [19]–[22]. [19] show that multi-task models which predict each annotator’s label separately outperform majority-vote aggregation on several binary classification tasks. [20] propose jury learning, which explicitly selects a composition of annotators at inference time to reflect a desired social perspective. While these works model disagreement as a signal to preserve, they do not address the setting where disagreement arises from differing levels of expertise on a factual task with a single correct answer, which is the setting we consider.

A related line of work addresses label noise in deep learning more broadly [8]–[10], and recent work has extended noise-robust training to the LLM fine-tuning setting [23]–[26]. These approaches typically assume a fixed noise structure, require external guidance or a clean reference set to identify noisy samples, and do not model annotator identity. In contrast, REALM requires neither: it learns annotator reliability unsupervised as part of training, with no access to ground-truth labels at any stage, and explicitly models the per-annotator source of noise rather than treating it as a property of individual examples. A separate line of work studies robustness to corrupted or adversarially perturbed inputs [27], [28]; this targets worst-case corruption of the input rather than the annotator-level label noise REALM addresses, and is orthogonal to our setting.

2.2 Learning from Suboptimal Demonstrators↩︎

Imitation learning from heterogeneous and suboptimal demonstrators is a well-studied problem in sequential decision-making. Standard behavioral cloning [29] treats all demonstrators as equally expert, which leads to a degraded policy when the dataset contains suboptimal behavior. Several methods address this by requiring additional supervision, such as confidence scores over demonstrations [30], access to a subset of expert demonstrations [31], or knowledge of environment dynamics [32].

A key insight motivating REALM is that annotator identity alone, without any quality labels, can be sufficient to recover reliable expertise estimates. A mixture-based demonstrator model captures this intuition naturally: the observed label is modeled as a convex combination of the optimal model’s prediction and a uniform random guess, weighted by the annotator’s (learned) expertise. This formulation was introduced in the sequential decision-making literature and shown to allow unsupervised recovery of demonstrator quality while simultaneously improving the learned policy [11], [12]. We adapt and extend this framework to the LLM fine-tuning setting, where supervision is token-level and expertise varies across tasks in a structured multi-task setting. To our knowledge, this mixture-based approach has not previously been applied to language model training.

2.3 Supervised Fine-Tuning and Alignment of Language Models↩︎

Supervised fine-tuning on human-annotated demonstrations is a standard first step in aligning large language models with human intent [1]–[4]. In practice, the annotation pipelines used to construct SFT datasets involve multiple crowdworkers with varying levels of domain expertise, task familiarity, and attention [1], [4]. Despite this, the vast majority of SFT pipelines aggregate annotations via majority vote or simple averaging before training, discarding information about annotator identity and quality. Recent work has shown that annotation quality matters far more than quantity for SFT: a small set of carefully curated examples can match or exceed models trained on orders of magnitude more data [7], and instruction data can even be generated without human annotation altogether [33]. This underscores the practical importance of methods that can automatically identify and down-weight low-quality annotations at training time.

Reinforcement learning from human feedback [1], [4], [34] addresses annotator disagreement to some extent through preference modeling, but requires pairwise comparisons rather than direct labels, and relies on a separate reward model stage that introduces its own sources of noise and bias. Methods that operate directly on the SFT stage, without requiring reward modeling or online feedback, are therefore of practical interest. REALM operates entirely within the SFT stage: it takes as input the same triples of (prompt, annotator, label) that standard SFT uses, and replaces the cross-entropy loss with an expertise-aware mixture objective. It requires no preference data, no clean reference set, no reward model, and no additional annotation beyond annotator identity. This makes it a lightweight and broadly applicable alternative to standard SFT when training data originates from multiple annotators of heterogeneous quality.

3 Proposed Algorithm↩︎

3.1 Problem Setup↩︎

We consider supervised fine-tuning of a language model on multiple-choice question answering under noisy annotations. Let \(\mathcal{Q}\) denote the space of prompts and \(\mathcal{A} = \{1, \ldots, K\}\) a finite set of answer choices. We assume access to a training set of triples \((Q, i, \tilde{Y})\), where \(Q \in \mathcal{Q}\) is a prompt, \(i \in \{1, \ldots, N\}\) indexes an annotator, and \(\tilde{Y} \in \mathcal{A}\) is a potentially noisy observed label. The ground-truth label \(Y^* \in \mathcal{A}\) is never observed during training.

Each annotator has an expertise level \(\beta_i \in [0, 1]\), interpreted as the probability that their label matches the ground truth. Standard SFT ignores annotator identity entirely, treating all \((Q, \tilde{Y})\) pairs as equally reliable, thereby absorbing the errors of low-expertise annotators directly into the model.

3.2 The REALM Objective↩︎

REALM models the observed label as a mixture between the model’s prediction and a uniform random guess, weighted by the annotator’s expertise. Let \(\theta\) denote the parameters of the language model and \(P_\theta(y \mid Q)\) its predicted probability for answer \(y\) given prompt \(Q\). The observed label distribution is: \[P(\tilde{Y} = y \mid Q, i) \;=\; \beta_i P_\theta(y \mid Q) \;+\; \frac{1 - \beta_i}{K}. \label{eq:mixture}\tag{1}\] When \(\beta_i = 1\) the annotator is a perfect expert; when \(\beta_i = 0\) they label uniformly at random.

Restricted softmax. For multiple-choice tasks, each answer choice corresponds to a single token in the model’s vocabulary (e.g.the tokens for a, b, c, d). Rather than computing a softmax over the full vocabulary of tens of thousands of tokens, we restrict the output distribution to the \(K\) answer tokens \(\{t_1, \ldots, t_K\}\), one per answer choice: \[P_\theta(y = k \mid Q) \;=\; \frac{\exp(z_{t_k})}{\sum_{k'=1}^{K} \exp(z_{t_{k'}})}, \label{eq:restricted95softmax}\tag{2}\] where \(z_v\) denotes the logit for vocabulary token \(v\) at the answer position in the generated sequence. This serves two purposes: it ensures \(P_\theta\) is a proper distribution over \(\mathcal{A}\) rather than being diluted across irrelevant tokens, and it aligns the model’s output space with the uniform term \(1/K\) in the mixture objective of Equation 1 , so that both components operate over the same \(K\)-class distribution.

Learning expertise. We parameterize each annotator’s expertise as \(\beta_i = \sigma(\omega_i)\), where \(\omega_i \in \mathbb{R}\) is a scalar trained jointly with the model parameters \(\theta\). The training objective is the negative log-likelihood of the observed labels under the mixture model:

\[\begin{align} \mathcal{L}(\theta, \boldsymbol{\omega}) &= -\sum_{(Q,\, i,\, \tilde{Y})} \log\Bigl[ \sigma(\omega_i) \cdot P_\theta(\tilde{Y} \mid Q) \notag\\ &\qquad\qquad+\; \frac{1 - \sigma(\omega_i)}{K} \Bigr]. \label{eq:loss} \end{align}\tag{3}\] where \(\boldsymbol{\omega} = \{\omega_i\}_{i=1}^{N}\). We optimise \(\theta\) and \(\boldsymbol{\omega}\) jointly using AdamW with separate learning rates, as the expertise scalars typically benefit from a larger step size than the model parameters. At convergence, \(\sigma(\omega_i)\) provides an interpretable, unsupervised estimate of annotator \(i\)’s reliability.

Relationship to standard SFT. When \(\omega_i \to +\infty\) for all \(i\), all \(\beta_i \to 1\) and \(\mathcal{L}\) reduces to the standard cross-entropy loss on the restricted softmax. REALM therefore strictly generalises SFT, recovering it in the high-expertise limit and progressively down-weighting annotator errors as expertise decreases.

3.3 Extension of REALM to the Multi-Task Setting↩︎

We extend REALM to a setting with \(M\) tasks (datasets), each with its own answer set \(\mathcal{A}^{(m)}\) of size \(K_m\). Let \(f^*\!: \mathcal{Q} \to \{1, \ldots, M\}\) be the ground-truth task assignment and write \(T = f^*(Q)\). In our experiments, \(f^*(Q)\) is always observed since we know which dataset each prompt originates from. Each annotator \(i\) is assigned a skill vector \(\boldsymbol{\omega}_i \in \mathbb{R}^M\), generalising the scalar \(\omega_i\) from Section 3.2 to one entry per task, where \(\omega_{i,m}\) captures the reliability of annotator \(i\) on task \(m\). Stacking these vectors as rows defines a learned \(N \times M\) expertise matrix \(\boldsymbol{\Omega}\), where \(\Omega_{i,m} = \omega_{i,m}\), and the expertise of annotator \(i\) on a prompt from task \(T\) is \(\sigma(\Omega_{i,T})\). The multi-task training objective is:

\[\begin{align} \mathcal{L}(\theta, \boldsymbol{\Omega}) &= -\sum_{(Q,\, i,\, \tilde{Y})} \log\Bigl[ \sigma\!\left(\Omega_{i,\, T}\right) \cdot P_\theta(\tilde{Y} \mid Q) \notag\\ &\qquad\qquad+\; \frac{1 - \sigma\!\left(\Omega_{i,\, T}\right)}{K_{T}} \Bigr]. \label{eq:multi95loss} \end{align}\tag{4}\] where \(K_T = K_{f^*(Q)}\) denotes the number of answer choices for task \(T\). In all settings the true matrix \(\boldsymbol{\Omega}\) is never provided; the model learns it entirely from the observed triples \((Q, i, \tilde{Y})\).

3.3.0.1 Per-task restricted softmax.

Because different tasks have different numbers of answer choices, we construct a separate set of answer token ids per task. Within each minibatch, examples are grouped by task so that the correct \(K_m\) is used for both the restricted softmax and the uniform term \(1/K_m\) in the mixture, ensuring that both components of the objective operate over the same \(K_m\)-class distribution.

4 Empirical Results↩︎

4.1 Experimental Setup↩︎

Datasets. We evaluate on five multiple-choice question answering benchmarks. For commonsense reasoning: OpenBookQA [13] (OBQA) and ARC [14], each with four answer choices (\(K{=}4\)); PIQA [15], a physical intuition benchmark with two choices (\(K{=}2\)); and RiddleSense [16], a commonsense riddle dataset with five choices (\(K{=}5\)). For biomedical reasoning: PubMedQA [17], a yes/no/maybe question answering dataset with three choices (\(K{=}3\)). These datasets span a range of task difficulties, domain types, and answer-set sizes, making them well-suited for studying how annotation noise and expertise structure interact with model performance. In the multi-task setting we combine three datasets simultaneously; single-task experiments are conducted on each dataset independently.

Implementation Details. We fine-tune three sizes of Flan-T5 [2]: small (80M), base (250M), and large (780M parameters). All models are fine-tuned using AdamW with a cosine learning rate schedule, a model learning rate of \(5{\times}10^{-5}\), a batch size of \(8\), and gradient accumulation over \(8\) steps, giving an effective batch size of \(64\). Training runs for \(200\) steps in all experiments. For REALM, the expertise scalars \(\boldsymbol{\omega}\) are optimised with a separate learning rate selected from \(\{0.5, 1.0\}\) based on validation performance, as the expertise parameters are low-dimensional and benefit from faster adaptation than the model parameters \(\theta\). All experiments are run on NVIDIA A6000 GPUs. All results are reported as the mean and standard deviation over five random seeds.

Baselines. We compare REALM against two baselines. Oracle SFT trains on ground-truth labels and serves as an upper bound; it is never achievable in practice since true labels are unobserved during training. Noisy SFT trains on noisy labels using standard cross-entropy, ignoring annotator identity entirely. This reflects the standard pipeline used in practice when annotations are aggregated without quality control.

Simulated Annotation. Since we do not have access to a real multi-annotator dataset with known ground-truth expertise levels, we simulate noisy annotations using the mixture model in Equation 1 . We use \(N{=}3\) annotators in all experiments. In the single-task setting, we evaluate three annotator expertise distributions: Dist.  (\(\beta{=}[0.9, 0.1, 0.1]\)), where one reliable annotator is outnumbered by two near-random ones; Dist.  (\(\beta{=}[0.9, 0.9, 0.1]\)), where two reliable annotators form a majority; and Dist.  (\(\beta{=}[0.9, 0.5, 0.1]\)), where expertise is spread uniformly from high to low. These configurations vary the proportion of reliable annotators, from a single reliable annotator (Dist.  ) to a majority (Dist.  ), allowing us to assess REALM across a wide range of annotation quality regimes.

Noise Types. In the standard uniform noise model assumed by the mixture objective, when an annotator makes an error, the wrong label is drawn uniformly at random over the \(K{-}1\) incorrect classes. Beyond this, we evaluate under two additional noise types that violate this assumption. In asymmetric noise, the wrong label concentrates on the cyclically adjacent class \((t{+}1) \bmod K\), which receives a weight of \(c{=}4\) times that of the remaining wrong classes. Concretely, with \(K{=}4\) the adjacent class receives approximately \(57\%\) of the wrong-label mass while the other two incorrect classes share the remaining \(43\%\). The marginal accuracy \(\beta_i\) is preserved, but the wrong-label distribution is no longer uniform, violating REALM’s generative assumption. In systematic noise, each annotator \(i\) deterministically emits class \((t + i + 1) \bmod K\) whenever they make an error, where \(t\) is the true class index, so wrong labels are fully predictable from annotator identity and the true label. Together, the three noise types allow us to assess REALM both under its assumed generative model and under increasingly severe forms of distributional mismatch.

Multi-Task Setup. In the multi-task setting we jointly train on three datasets simultaneously with \(N{=}3\) annotators and \(M{=}3\) tasks, yielding a square expertise matrix \(\boldsymbol{\Omega} \in \mathbb{R}^{3 \times 3}\) where each entry \(\Omega_{i,m}\) represents the reliability of annotator \(i\) on task \(m\). We evaluate three structured expertise configurations: Dist.  , in which annotator \(i\) is reliable only on task \(i\) (\(\beta_{i,i}{=}0.9\), \(\beta_{i,j}{=}0.1\) for \(j \neq i\)); Dist.  , the inverse pattern in which annotator \(i\) is reliable on all tasks except task \(i\) (\(\beta_{i,i}{=}0.1\), \(\beta_{i,j}{=}0.9\) for \(j \neq i\)); and Dist.  , in which annotator \(i\) has expertise \(\beta_{i,j} = [0.9, 0.5, 0.1][(j{-}i) \bmod M]\) on task \(j\), so annotator \(i\) is most reliable on their own task \(i\) (\(\beta{=}0.9\)), intermediate on task \((i{+}1) \bmod M\) (\(\beta{=}0.5\)), and least reliable on task \((i{+}2) \bmod M\) (\(\beta{=}0.1\)).

4.2 Single-Task Results↩︎

Results under Uniform Noise. Table 1 reports test accuracy under uniform noise across all datasets, model sizes, and expertise distributions. realm outperforms noisy in all settings, with gains that grow with annotation difficulty. Improvements are largest under Dist.  , where only one annotator is reliable and the two unreliable annotators dominate the training signal. Here noisy degrades severely as it absorbs the majority noise directly, while realm recovers substantially by learning to down-weight the two near-random annotators. Under Dist.  , the gains are more modest because the intermediate annotator (\(\beta{=}0.5\)) provides a partial signal that reduces the damage of naive aggregation, leaving less room for the mixture correction to help. Under Dist.  , the gap narrows further still, as the two reliable annotators provide a strong majority signal that noisy can exploit without requiring expertise weighting, and realm still wins consistently on OBQA, Riddle, and PIQA.

Robustness to Noise Type. Table 2 reports results under Dist.  across uniform, asymmetric, and systematic noise types. realm remains robust under all three conditions, consistently outperforming noisy despite both asymmetric and systematic noise violating the uniform assumption of the mixture objective. The key insight is that even under misspecified noise, the observed labels still carry information about annotator expertise: unreliable annotators produce systematically worse labels regardless of the noise structure, and realm is able to identify and down-weight them. Asymmetric noise introduces a directional bias in the wrong-label distribution but preserves this signal, and realm performs comparably to the uniform setting. Systematic noise is the hardest case, as each annotator’s errors follow a fixed deterministic offset that partially masks their true reliability, reflected in the average \(\Delta\) dropping to \(+5.39\%\) (Small) and \(+5.13\%\) (Base) compared to \(+6.65\%\) and \(+10.24\%\) under uniform noise.

Model Size Effect. The \(\Delta\) columns in Tables 1 and 2 reveal a consistent scaling trend across most settings: the average gain under Dist.  grows from \(+6.65\%\) to \(+10.52\%\) to \(+17.25\%\) across Small, Base, and Large under uniform noise, and from \(+7.02\%\) to \(+10.50\%\) to \(+19.44\%\) under asymmetric noise. This suggests that larger models have greater capacity to exploit the expertise signal provided by the mixture objective, translating annotator reliability estimates into larger accuracy improvements. Under systematic noise the trend is less pronounced, likely because the deterministic error structure limits how much additional capacity can help when the reliability signal itself is partially masked.

4.3 Multi-Task Results↩︎

Table 3 reports combined accuracy on ARC, OBQA, and Riddle in the multi-task setting, where annotator expertise varies across tasks according to a learned \(3{\times}3\) matrix \(\boldsymbol{\Omega}\). In this setting, realm outperforms noisy in every configuration across all model sizes, expertise distributions, and noise types. As in the single-task setting, the largest gains occur under Dist.  , where only one annotator is reliable per task. Here realm outperforms noisy by up to \(11.75\%\) (Large, Asymmetric: \(60.44\%\) vs.\(48.69\%\)), approaching Oracle accuracy in several cases. Under Dist.  and Dist.  the gains are more modest but remain consistent, suggesting that realm reliably recovers the structured per-task expertise pattern even when the signal is weaker. Per-dataset breakdowns are provided in the Appendix, where the gains of realm are consistent across ARC, OBQA, and Riddle individually.

4.4 Ablation Study↩︎

Expertise Learning Rate. Table 4 ablates the expertise scalar learning rate \(\eta_\omega\) with the model learning rate fixed at \(5{\times}10^{-5}\). Low values (\(\eta_\omega \leq 0.05\)) cause severe degradation, most notably on PIQA (\(40.91\%\) at \(\eta_\omega{=}0.01\)), because the expertise scalars adapt too slowly to influence training before the model has already absorbed the noise. Performance stabilises at \(\eta_\omega \geq 0.1\) for most datasets, with PIQA requiring \(\eta_\omega{=}0.5\) to fully recover, as the expertise scalars must adapt quickly enough to influence the model before it converges to a noisy solution.

Number of Annotators. Table 5 reports accuracy as the number of annotators grows from \(N{=}3\) to \(N{=}10\) under Dist.  . realm outperforms noisy across all datasets and annotator counts, demonstrating that the method scales reliably as the annotator pool grows. As \(N\) increases, annotator expertise values are placed on a uniform grid from \(\beta{=}0.9\) down to \(\beta{=}0.1\), i.e. \(\beta_i = 0.9 - i \cdot \tfrac{0.8}{N-1}\) for \(i = 0, \ldots, N-1\), introducing a larger number of intermediate-reliability annotators with smaller pairwise differences in expertise that are harder to distinguish from one another. This explains the modest drop in average \(\Delta\) from \(+5.93\%\) (\(N{=}3\)) and \(+5.98\%\) (\(N{=}5\)) to \(+5.01\%\) (\(N{=}10\)). Despite this, the gains remain consistent across all annotator counts, confirming that realm is robust to larger annotator pools.

4.5 Learning Curves↩︎

Figure 1 shows test accuracy over training steps under Dist.  for all three noise types. Under all three conditions, realm converges smoothly and remains consistently above noisy, while noisy degrades on PIQA as training progresses by absorbing the dominant noise signal from the two unreliable annotators. The separation is most pronounced under asymmetric noise, where noisy collapses more severely, while under systematic noise the gap is smaller but still consistent. Across all noise types, realm exhibits noticeably lower variance than noisy, indicating that jointly learning annotator expertise stabilises training even under distributional mismatch between the noise model and the true label generation process.

5 Conclusion↩︎

We presented REALM, a mixture-based fine-tuning objective that jointly learns language model parameters and per-annotator expertise scalars entirely unsupervised. By modeling each observed label as a convex combination of the model’s prediction and a uniform random guess, REALM automatically identifies and down-weights unreliable annotators without any supervision beyond annotator identity. Experiments on five question answering benchmarks across three model sizes, three noise types, and multiple expertise distributions show that REALM consistently outperforms naive noisy SFT in the large majority of settings, with gains that grow with model capacity and persist even when the noise type violates the model’s generative assumptions. In the multi-task extension, a learned expertise matrix captures per-task annotator reliability and further improves performance across all configurations.

6 Limitations↩︎

Fixed expertise per annotator. REALM assigns a single scalar expertise value to each annotator, constant across all prompts. In practice, an annotator may be reliable on certain question types or domains while performing poorly on others, even within the same dataset. A natural generalisation is to model expertise as a function of the input, \(\rho_i(Q) = \sigma(\langle \boldsymbol{\omega}_i, \phi(Q) \rangle)\), where \(\phi(Q) \in \mathbb{R}^d\) is a learned prompt embedding and \(\boldsymbol{\omega}_i \in \mathbb{R}^d\) is a per-annotator skill vector. This allows expertise to vary continuously across the input space, and subsumes both the single-task scalar (\(d{=}1\)) and the multi-task matrix lookup (when \(\phi(Q)\) is the one-hot task encoding) as special cases. We leave the full exploration of this generalisation to future work.

Multiple-choice tasks. The restricted softmax formulation requires a fixed, discrete set of answer tokens, limiting the current method to multiple-choice tasks. Extending REALM to open-ended generation, where there is no natural \(K\)-class output space, requires a different notion of annotator expertise and a modified mixture objective.

Efficient and Distributed Fine-Tuning. Fine-tuning large language models at scale introduces practical challenges beyond data quality, including communication overhead, privacy constraints, and computational efficiency. In many real-world annotation pipelines, training data is collected from multiple independent sources or organizations, each with its own data distribution and annotation practices, mirroring the federated learning setting where data heterogeneity across clients is a central challenge. Recent work has extended parameter-efficient fine-tuning methods to federated [35] and fully decentralized settings [36], with explicit mechanisms for handling non-i.i.d.data distributions and convergence guarantees. The data heterogeneity problem in these settings is conceptually related to the annotator heterogeneity problem studied in this paper: in both cases, a model must be trained on data from multiple sources of varying quality without access to a unified gold standard. REALM addresses this at the annotation level by learning per-source reliability weights, and combining expertise-aware objectives with federated optimization represents a natural direction for future work.

7 Additional Results↩︎

7.1 Results under Dist. 2⃝ and Dist. 3⃝↩︎

Tables 6 and 7 report results under the remaining two expertise distributions. Under Dist.  , where two reliable annotators form a majority, gains are modest (avg \(\Delta \leq +4\%\)) since noisy can already exploit the strong majority signal, with PubMedQA reversing under systematic noise at Base and Large model sizes. Under Dist.  , gains are larger and asymmetric noise yields the highest average \(\Delta\) across all model sizes, while systematic noise again produces the smallest gains with occasional reversals on PubMedQA. Overall, realm outperforms noisy in the large majority of settings across both distributions, noise types, and model sizes.

7.2 Per-Dataset Multi-Task Results↩︎

Table 8 provides the per-dataset breakdown of the combined multi-task results reported in Table 3. realm outperforms noisy on the large majority of individual dataset and configuration combinations, confirming that the gains observed in the combined accuracy are not driven by a single dataset. The most consistent and largest gains appear on OBQA and Riddle, which have more answer choices (\(K{=}4\) and \(K{=}5\) respectively), giving the mixture objective a stronger signal to discriminate reliable from unreliable annotators. ARC shows smaller but still consistent gains, while a small number of reversals occur on ARC under Dist. #2 for the Small model, where the noise level is low enough that noisy benefits from the larger shared training set.

References↩︎