Abstract

Temperature scaling is the dominant post-hoc calibration method in modern deep learning. Its theoretical justification rests on an assumption that is rarely stated explicitly: that ground-truth labels are one-hot and deterministic. In practice, labels are frequently soft, crowd-sourced, or genuinely distributional, reflecting real disagreement among human annotators rather than annotation noise. We study whether temperature scaling retains its calibration properties when this assumption is violated, and whether any resulting degradation depends on model scale. Using CIFAR-10H and ChaosNLI, two publicly available datasets with human-annotated soft label distributions, we evaluate three model scales per modality under both hard one-hot and soft distributional label targets. Across all nine configurations we find a positive soft-label calibration gap: temperature scaling calibrated on hard labels consistently underperforms an oracle calibrated directly on soft labels, with Brier Score gaps ranging from 0.002 to 0.134. The gap grows monotonically with model scale in the vision domain and on the SNLI-derived split of ChaosNLI, and is substantially larger in the language domain (mean gap 0.079) than in vision (mean gap 0.003). A scale-ordering reversal on the MNLI-derived split remains after matched-domain training; we treat it as inconclusive for the scale hypothesis and attribute it primarily to near-chance accuracy on that split. As a second post-hoc baseline, multiclass isotonic regression yields the same qualitative conclusion: positive soft-label gaps in all nine configurations, and larger gaps in language than in vision. These findings suggest that calibration protocols built on majority-vote labels systematically misstate model reliability wherever label ambiguity is structural, with direct consequences for deployment in safety-critical settings.

Keywords: calibration; temperature scaling; soft labels; label ambiguity; Expected Calibration Error; Brier Score; isotonic regression; model scale; uncertainty quantification.

1 Introduction↩︎

A probabilistic classifier is well-calibrated when its predicted confidence matches empirical accuracy: among all predictions assigned confidence \(p\), approximately a fraction \(p\) should be correct [1]. In safety-critical applications, a miscalibrated model that is confidently wrong poses a greater risk than an uncertain model that correctly signals its own limitations. The consequences of overconfident predictions are well documented in medical imaging, content moderation, and judicial risk assessment.

Temperature scaling [2] is the most widely deployed post-hoc calibration method for neural networks. A single scalar \(T\) is fit by minimizing negative log-likelihood on a held-out validation set, and logits are divided by \(T\) before the softmax at inference time. The method adds no parameters and requires no retraining, which accounts for its near-universal adoption. Its limitation, which we measure in this paper, is that its optimization target implicitly assumes ground-truth labels are one-hot and deterministic.

This assumption fails whenever a labeling task involves genuine human disagreement, such that the correct answer is not a single class but a distribution over plausible answers [3]. Two datasets make this disagreement explicit. CIFAR-10H [4] provides per-image soft label distributions for the CIFAR-10 test set, derived from a mean of 51 human annotations per image. ChaosNLI [5] provides 100 human judgments per instance for natural language inference examples that exhibit substantial semantic ambiguity. A model that assigns high confidence to a single class when human annotators are evenly split is not calibrated in any meaningful sense, even when its predicted class coincides with the majority vote.

We conduct a controlled measurement study to test whether temperature scaling, calibrated against hard one-hot labels, remains an effective calibration strategy when evaluated against soft, distributional targets, and whether any degradation scales with model capacity. We additionally compare against multiclass isotonic regression [6] under the same hard and soft fitting protocols.

1.1 Hypotheses↩︎

We test three pre-specified hypotheses.

• H1 (Gap existence). Temperature scaling calibrated on hard labels yields a strictly worse Brier Score against the soft label distribution than an oracle calibrated directly on soft labels. We call this difference the soft-label calibration gap (Section 3.4).

• H2 (Scale dependence). The soft-label calibration gap increases monotonically with model scale within each dataset, consistent with the finding that larger models tend toward higher confidence [2].

• H3 (Modality dependence). The soft-label calibration gap is larger in the language domain (ChaosNLI) than in the vision domain (CIFAR-10H), reflecting higher rates of genuine semantic ambiguity in natural language inference relative to object classification.

1.2 Contributions↩︎

• We formalize the soft-label calibration gap and provide a protocol for estimating it with existing soft-label datasets (Section 3).

• We report a systematic cross-scale measurement across two modalities and two soft-label resources (Section 5).

• We analyze a scale-ordering anomaly on ChaosNLI-M under matched-domain training and evaluation (Section 5.3).

• We show that the gap persists under isotonic regression, a second post-hoc calibrator (Section 5.5).

• We discuss implications for calibration practice in ambiguous- and resource-constrained label settings (Section 6).

2 Related Work↩︎

2.1 Calibration of Neural Networks↩︎

Guo et al.[2] established that modern deep networks are substantially more overconfident than earlier architectures and proposed temperature scaling as an effective post-hoc correction. Subsequent work has proposed isotonic regression [6], Platt scaling [7], and Dirichlet calibration [8] as alternatives. Minderer et al.[9] found that Vision Transformers tend to be better calibrated in-distribution than convolutional architectures. Bai et al.[10] showed analytically that overconfidence is not solely a consequence of overparameterization. Kumar et al.[11] showed that ECE tends to underestimate true miscalibration due to binning artifacts, motivating our use of Brier Score as a complementary metric.

2.2 Soft Labels and Human Disagreement↩︎

Work on learning from crowds [12] and annotation disagreement [3] has established that human label variation carries information about task ambiguity. Peterson et al. [4] introduced CIFAR-10H. Nie et al.[5] introduced ChaosNLI to surface genuine semantic ambiguity that majority-vote labeling obscures. Their use as calibration benchmarks under distributional targets is the specific focus of this paper.

2.3 Calibration Under Distribution Shift↩︎

Ovadia et al.[13] showed that calibration degrades under dataset shift even where accuracy is partially preserved. Guillory et al.[14] found that confidence-based accuracy estimators degrade under natural distribution shift. Wiles et al.[15] showed that larger models and larger pretraining corpora tend to improve both OOD accuracy and calibration under standard protocols. None of this work examines the case in which the shift occurs in the label distribution rather than the input distribution.

3 Problem Formulation↩︎

3.1 Notation↩︎

Let \(\mathcal{X}\) be the input space and \(\mathcal{Y}=\{1,\ldots,K\}\) the label space. A classifier \(f:\mathcal{X}\rightarrow\Delta(\mathcal{Y})\) maps inputs to probability distributions over classes. A hard-label dataset \(\mathcal{D}_{\mathrm{hard}}=\{(x_i,y_i)\}\) pairs each input with a single class label \(y_i\in\mathcal{Y}\). A soft-label dataset \(\mathcal{D}_{\mathrm{soft}}=\{(x_i,q_i)\}\) pairs each input with a distribution \(q_i\in\Delta(\mathcal{Y})\) representing the empirical distribution of human annotations for instance \(x_i\).

3.2 Calibration Under Hard Labels↩︎

A classifier \(f\) is calibrated under hard labels if, for all \(p\in[0,1]\), \[\mathbb{P}\!\left(y=\arg\max f(x)\,\middle|\,\max f(x)=p\right)=p.\] ECE approximates this by partitioning predictions into \(M\) bins [2]: \[\mathrm{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\left|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\right|.\] Temperature scaling finds \(T^\star\) minimizing negative log-likelihood on \(\mathcal{D}_{\mathrm{val}}\), where \(p_T(y\mid x)=\mathrm{softmax}(\mathrm{logits}(x)/T)\) [2]: \[T^\star=\arg\min_T\; -\frac{1}{|\mathcal{D}_{\mathrm{val}}|} \sum_{(x,y)\in\mathcal{D}_{\mathrm{val}}} \log p_T(y\mid x).\]

3.3 Calibration Under Soft Labels↩︎

A classifier is soft-calibrated if its predictive distribution matches the human annotation distribution in expectation. The Brier Score [16] is a strictly proper scoring rule over soft targets: \[\mathrm{BS}=\frac{1}{n}\sum_{i=1}^{n}\sum_{k=1}^{K} \bigl(f_k(x_i)-q_{ik}\bigr)^2.\]

3.4 The Soft-Label Calibration Gap↩︎

An analogous gap is defined for isotonic regression by replacing the temperature maps with calibrators fit on hard validation labels and on soft oracle labels, respectively.

4 Experimental Design↩︎

4.1 Datasets↩︎

We use two publicly available soft-label datasets. CIFAR-10H [4] provides a mean of 51 human annotations per image for all 10,000 CIFAR-10 test images. ChaosNLI [5] provides 100 human judgments per instance for NLI examples from SNLI (ChaosNLI-S) and MNLI matched (ChaosNLI-M).

4.2 Models↩︎

For vision we use ResNet-18 (11M), ResNet-50 (25M), and ResNet-101 (44M) [17], trained from scratch on CIFAR-10 with random crop, horizontal flip, SGD with cosine-annealed learning rate, and 30 epochs (the recipe used in our released code). For language we use DistilBERT-base-uncased [18], BERT-base-uncased, and BERT-large-uncased [19]. Each language run starts from a domain-matched NLI checkpoint (SNLI for ChaosNLI-S; MNLI for ChaosNLI-M) and is adapted for one epoch on a 10,000-example training subset with learning rate \(2\times10^{-5}\) and AdamW. For BERT-large on SNLI we use an independently fine-tuned checkpoint starting from base bert-large-uncased (not from an MNLI checkpoint), so the ChaosNLI-S condition is not confounded by MNLI pretraining.

4.3 Calibration Protocol↩︎

For each model: (1) train or adapt on hard labels; (2) fit \(T_{\mathrm{hard}}\) on a held-out hard-label validation split by minimizing Equation 2 with L-BFGS; (3) fit \(T_{\mathrm{oracle}}\) on the first half of the soft-label test set by minimizing Equation 3; (4) evaluate both on the held-out second half; (5) compute ECE (15 equal-width bins) and Brier Score against hard and soft targets. For isotonic regression we fit per-class one-vs-rest isotonic maps [6] on softmax scores, using the same hard-validation and soft-oracle splits, and renormalize the calibrated scores to a probability simplex. Every configuration is repeated with seeds 42, 123, and 456; we report mean and standard deviation.

5 Results↩︎

5.1 Vision: CIFAR-10H↩︎

2.8pt

Table 1 reports vision results. The soft-label calibration gap is positive for all three models (H1 supported) and increases with scale from 0.002 to 0.003 (H2 supported in vision). The absolute magnitude is modest: the largest gap is roughly 3% of the TS-Hard Brier Score for ResNet-101. The \(T_{\mathrm{oracle}}/T_{\mathrm{hard}}\) ratio ranges from 1.26 to 1.32, consistent with comparatively low annotator disagreement in CIFAR-10H.

5.2 Language: ChaosNLI↩︎

2.6pt

Table 2 reports language results. On ChaosNLI-S the gap increases monotonically with scale: 0.045, 0.050, 0.053 (H2 supported). The \(T_{\mathrm{oracle}}/T_{\mathrm{hard}}\) ratio is about 2.2 to 3.5 in language versus 1.2 to 1.3 in vision. For BERT-large on ChaosNLI-S the gap of 0.053 is about 17% of the TS-Hard Brier Score.

5.3 The ChaosNLI-M Anomaly↩︎

On ChaosNLI-M the scale ordering reverses: DistilBERT (0.134) \(>\) BERT-base (0.119) \(>\) BERT-large (0.074). Training and evaluation are matched-domain (MNLI training, ChaosNLI-M evaluation), so the reversal is not explained by an SNLI-to-MNLI train/test mismatch. We also no longer attribute it to an MNLI-pretrained BERT-large SNLI confound: that condition now uses an independent SNLI checkpoint with \(T_{\mathrm{hard}}\approx0.980\), and the ChaosNLI-M ordering is unchanged. The remaining confound is that all three models achieve only 50–53% accuracy on a three-class task (near chance). At this accuracy level, confidence estimates can spuriously resemble soft-label alignment. We treat ChaosNLI-M as inconclusive for H2.

5.4 Temperature Ratios↩︎

\(T_{\mathrm{oracle}}\) exceeds \(T_{\mathrm{hard}}\) in all nine configurations, from 1.26 (ResNet-18) to 3.48 (DistilBERT on ChaosNLI-M). This ratio measures how much smoothing hard-label calibration leaves unrealized and is consistently larger in language.

5.5 Isotonic Regression Baseline↩︎

Table 3 compares gaps under temperature scaling and isotonic regression. H1 holds under both methods in all nine configurations. Language ISO gaps are at least as large as TS gaps; vision gaps remain small under both. The soft-label gap is therefore not an artifact of temperature scaling alone.

5.6 Summary of Hypothesis Outcomes↩︎

H1 is supported across all nine configurations for temperature scaling and for isotonic regression. Gap-to-standard-deviation ratios for temperature scaling exceed 8 in five of the seven configurations whose reported standard deviation does not round to 0.000 at three decimals, and exceed 18 in four language cases. H2 is supported in vision and on ChaosNLI-S; ChaosNLI-M is inconclusive. H3 is supported: the mean language gap (0.079) is about 28 times the mean vision gap (0.003) when computed from unrounded means.

6 Implications↩︎

6.1 For Calibration Methodology↩︎

Calibration validation sets built on majority-vote labels systematically underestimate the smoothing required to match soft human targets. Where feasible, practitioners should collect repeated annotations and fit calibration parameters against the resulting distribution. Our oracle and isotonic results show that this requires no change to model architecture or training, only to the calibration step.

6.2 For Large Model Deployment↩︎

The soft-label calibration gap grows with model scale in vision and on ChaosNLI-S. Larger models are not automatically better calibrated under soft-label evaluation after hard-label temperature scaling. For BERT-large on ChaosNLI-S, hard-label temperature scaling yields Brier Score 0.318 against soft labels versus 0.265 under the soft oracle (about 20% relative degradation).

6.3 For Resource-Constrained Deployment↩︎

The stakes are highest where annotation infrastructure is weakest. In many clinical settings across sub-Saharan Africa, obtaining many independent annotations per case is impractical. A model that collapses clinician disagreement into an overconfident point prediction conceals exactly the uncertainty needed for appropriate caution. We designed this study to be reproducible on freely available compute and public data because practitioners facing this problem most acutely often have the least access to large-scale infrastructure.

7 Limitations↩︎

We highlight four limitations. First, our scale range (66M–340M for language; 11M–44M for vision) does not extend to frontier-scale models, and we make no extrapolation claim. Second, language adaptation uses a 10,000-example subset and one epoch for computational feasibility; full-corpus multi-epoch fine-tuning may change absolute Brier Scores while leaving the qualitative gap intact. Third, ChaosNLI-M remains inconclusive for H2 because of near-chance accuracy. Fourth, both datasets reflect annotation by English-speaking participants; generalization to other linguistic and cultural contexts requires further evidence.

8 Conclusion↩︎

Temperature scaling calibrated on hard labels does not fully calibrate models when the true label distribution is soft. This gap is consistent across all nine configurations, grows with model scale in vision and on ChaosNLI-S, and is roughly thirty times larger in language than in vision. The reversed ordering on ChaosNLI-M is better explained by near-chance accuracy than by a contradiction of the broader trend. The same qualitative pattern holds under isotonic regression.

Three directions follow: systematic comparison of additional calibration methods under soft-label evaluation; extension to larger models with high-accuracy matched-domain splits; and development of a practical soft-label calibration procedure usable with only a small number of annotations per instance.

Code: https://github.com/dogahwisdom/temperature-scaling-research

Acknowledgments↩︎

I am grateful to colleagues at BlackMatrix AI Research, Accra, for conversations that sharpened this work. All experiments were conducted on freely available compute with no institutional funding.

References↩︎