1 Introduction↩︎

Deep learning has achieved great success in many machine learning problems in the past decades, especially where large-scale labeled datasets are present. In real-world applications, however, manually labeling sufficient data is time-consuming and labor-exhaustive. To reduce the requirement for labeled data, semi-supervised learning (SSL) improves the data efficiency of deep models by learning from a few labeled samples and a large number of unlabeled samples [1]–[4]. Among them, self-training is an effective approach to deal with the lack of labeled data. Typical self-training methods [2], [5] assign pseudo labels to unlabeled samples with the model’s predictions and then iteratively train the model with these pseudo labeled samples as if they were labeled examples.

Although self-training has achieved great advances in benchmark datasets, they still exhibit large training instability and extreme performance imbalance across classes. For instance, the accuracy of FixMatch [5], one of the state-of-the-art self-training methods, fluctuates greatly when trained from scratch (see Figure [fig:scratch95loss95and95acc]). Though its performance will gradually recover after a sudden sharp drop, this is still not expected, since pre-trained models are more often adopted [4], [6], [7] are improve data efficiency, and the performance of pre-trained models is difficult to recover after a drastic decline due to catastrophic forgetting [8]. Besides, although FixMatch improves the average accuracy, it also leads to the Matthew effect, i.e., the accuracy of well-behaved categories is further increased while that of poorly-behaved ones is decreased to nearly zero (see Figure 3). This is also not expected, since most machine learning models prefer performance balance across categories, even when the class imbalance exists in the training data [9]. The above findings are caused by the bias between the pseudo labeling function with the unknown target labeling function. Training with biased and unreliable pseudo labels has the chance to accumulate errors and ultimately lead to performance fluctuations. And for those poorly-behaved categories, the bias of the pseudo labels gets worse and will be further enhanced as self-training progresses, ultimately leading to the Matthew effect.

We delved into the bias issues arising from the self-training process and found that they can be briefly grouped into two kinds: (1) Data bias, the bias inherent in the SSL tasks; (2) Training bias, the bias increment brought by self-training with incorrect pseudo labels. In this regard, we present Debiased Self-Training (DST), a novel approach to decrease the undesirable bias in self-training. Specifically, to reduce the training bias, the classifier head is only trained with clean labeled samples and no longer trained with unreliable pseudo-labeled samples. In other words, the generation and utilization of pseudo labels are decoupled to mitigate bias accumulation and boost the model’s tolerance to biased pseudo labels. Further, to decrease the data bias which cannot be calculated directly, we turn to estimate the worst case of training bias that implicitly reflects the data bias. Then we optimize the representations to decrease the worst-case bias and thereby improve the quality of pseudo labels.

The contributions of this work are summarized as follows: (1) We systematically identify the problem and analyze the causes of self-training bias in semi-supervised learning. (2) We propose DST, a novel approach to mitigate the self-training bias and boost the stability and performance balance across classes, which can be used as a universal add-on for mainstream self-training methods. (3) We conduct extensive experiments and validate that DST achieves an average boost of \(6.3\%\) against state-of-the-art methods on standard datasets and \(18.9\%\) against FixMatch on \(13\) diverse tasks.

2 Related Work↩︎

2.1 Self-training for semi-supervised learning↩︎

Self-training [1], [2], [10], [11] is a widely-used approach to utilize unlabeled data. Pseudo Label [2], one popular self-training method, iteratively generates pseudo labels and utilizes them with the same model. However, this paradigm suffers from the problem of confirmation bias [12], where the learner struggles to correct its own mistakes when learning from inaccurate pseudo labels. The bias issue is also mentioned in DebiasMatch [13] where they define the bias as the quantity imbalance for each category. Note that the bias in our paper refers to the deviation between the pseudo labeling function and the ground truth labeling function, which is a more essential problem existing in most self-training methods. Recent works mainly tackle this bias issue from the following two aspects.

Generate higher-quality pseudo labels. MixMatch [14] averages predictions from multiple augmentations as pseudo labels. ReMixMatch [15], UDA [16], and FixMatch [5] adopt confidence thresholds to generate pseudo labels on weakly augmented samples and utilize these pseudo-labels as annotations for strongly augmented samples. Dash [17] and FlexMatch [18] dynamically adjust the thresholds in a curriculum learning manner. Label Propagation methods [19], [20] assign pseudo labels with the density of the local neighborhood. DASO [21] blends the confidence-based pseudo labels and density-based pseudo labels differently for each class. Meta Pseudo Labels [22] proposes to generate pseudo labels with a meta learner. Different from the above methods that manually design specific criteria to improve the quality of pseudo labels, we estimate the worst case of self-training bias and adversarially optimize the representations to improve the quality of pseudo labels automatically.

Improve tolerance to inaccurate pseudo labels. To mitigate the confirmation bias, existing methods maintain a mismatch between the generation and utilization of pseudo labels. Temporal Ensembling [23] and Mean Teacher [3] generate pseudo labels from the average of previous predictions or an exponential moving average of the model, respectively. Noisy Student [24] assigns pseudo labels by a fixed teacher from the previous round. Co-training [25], MMT [26], DivideMix [27] and Multi-head Tri-training [28] introduce multiple models or classifier heads and learn in an online mutual-teaching manner. In these methods, each classifier head is still trained with potentially incorrect pseudo labels generated by other heads. In contrast, in our method, the classifier head that generates pseudo labels is never trained with pseudo labels, leading to better tolerance to inaccurate pseudo labels (Table 2).

2.2 Self-supervised learning for semi-supervised learning↩︎

Self-supervised methods [6], [29] are also used on unlabeled data to improve the model with few labeled samples, either in the pre-training stage [4], [30] or in the downstream tasks [31], [32]. However, the training of self-supervision usually relies on big data and heavy computation, which is not feasible in most applications. Besides, although these methods avoid the use of unreliable pseudo labels, it is difficult for them to learn task-specific information from unlabeled data for better performance.

2.3 Adversarial training for semi-supervised learning↩︎

Some works introduce adversarial training [33] into semi-supervised learning. A line of works [34]–[37] exploit fake samples from the generator by labeling them with a new “generated” class and forcing the discriminator to output class labels. Another line of works use adversarial training to construct adversarial samples [38], e.g., VAT [39] injects additive noise into input, VAdD [40] introduces adversarial Dropout [41] layers and RAT [42] expands the noise in VAT into a set of input transformations. These methods aim to impose a local smoothness on the model and do not involve training with pseudo labels. In contrast, in our method, the goal of the adversarial training is to estimate the worst case of pseudo labeling and then avoid such cases (Section 4.2).

3 Analysis of Bias in Self-Training↩︎

In this section, we provide some analysis of where the bias in self-training comes from. Let \(\mathcal{P}\) denote a distribution over input space \(\mathcal{X}\). For classification with \(K\) classes, let \(P^k\) denote the class-conditional distribution of \(\mathbf{x}\) conditioned on ground truth \(f^*(\mathbf{x})=k\). Assume that pseudolabeler \(f_{\text{pl}}\) is obtained via training a classifier on \(n\) labeled samples \(\widehat{P}_n\). Let \(\mathcal{M}(f_{\text{pl}})\triangleq \{\mathbf{x}: f_{\text{pl}}(\mathbf{x}) \neq f^*(\mathbf{x})\}\) denote the mistaken pseudolabeled samples. The bias in the self-training refers to the deviation between the learned decision hyperplanes and the true decision hyperplanes, which can be measured by the fraction of incorrectly pseudolabeled samples in any classes \(\mathcal{B}(f_{\text{pl}}) = \{ P^k(\mathcal{M}(f_{\text{pl}})) \}_{k=1}^{K}\) [43]. By analyzing self-training bias under different training conditions, we have several nontrivial findings.

The sampling of labeled data will largely influence the self-training bias. As shown in Figure 1, when the data sampling is different, the accuracy of the same category may vary dramatically. The reason is that the distances between different data points and the true decision hyperplanes are not the same, with some supporting data points closer and others far away. When there are few labeled data, there may be a big difference in the distances between supporting data of each category and the true decision hyperplanes, hence the learned decision hyperplanes will be biased towards some categories.

The pre-trained representations also affect the self-training bias. Figure 2 shows that different pre-trained representations lead to different category bias, even if the pre-trained dataset and the downstream labeled dataset are both identical. One possible reason is that the representations learned by different pre-trained models focus on different aspects of the data [44]. Therefore, the same data could also have different distances to the decision hyperplanes in the representation level with different pre-trained models.

Training with pseudo labels aggressively in turn enlarges the self-training bias on some categories. Figure [fig:training95bias] shows that after training with pseudo labels (e.g., using FixMatch), the performance gap for different categories greatly enlarges, with the accuracy of some categories increasing from \(60\%\) to \(80\%\) and that of some categories dropping from \(15\%\) to \(0\%\). The reason is that for well-behaved categories, the pseudo labels are almost accurate, hence using them for training could further reduce the bias. Yet for many poorly-behaved categories, the pseudo labels are not reliable, and the common self-training mechanism that uses these incorrect pseudo labels to train the model will further increase the bias, and fail to correct it back in the follow-up training. This results in the Matthew effect.

Based on the above observations, we divide the bias caused by self-training into two categories.

Data bias: the bias inherent in semi-supervised learning tasks, such as the bias of sampling and pre-trained representations on unlabeled data. Formally, data bias is defined as \(\mathcal{B}(f_{\text{pl}}(\widehat{P}_n, \psi_0))-\mathcal{B}(f^*)\) (blue area in Figure 3 (a)), where the pseudolabeler \(f_{\text{pl}}(\widehat{P}_n, \psi_0)\) is obtained from a biased sampling \(\widehat{P}_n\) with a biased parameter initialization \(\psi_0\).

Training bias: the bias increment brought by some unreasonable training strategies. Formally, training bias is \(\mathcal{B}(f_{\text{pl}}(\widehat{P}_n, \psi_0, \mathcal{S}))-\mathcal{B}(f_{\text{pl}}(\widehat{P}_n, \psi_0))\) (yellow area in Figure 3 (a)) where \(f_{\text{pl}}(\widehat{P}_n, \psi_0,\mathcal{S})\) is a pseudolabeler obtained with self-training strategy \(\mathcal{S}\).

Next we will introduce how to reduce training bias and data bias in self-training (red area in Figure 3 (a)).

4 Debiased Self-Training↩︎

In semi-supervised learning (SSL), we have a labeled dataset \(\mathcal{L}=\{(\mathbf{x}_i^l, {y}_i^l \}_{i=1}^{n_l}\) of \(n_l\) labeled samples and an unlabeled dataset \(\mathcal{U}=\{(\mathbf{x}_j^u) \}_{j=1}^{n_u}\) of \(n_u\) unlabeled samples, where the size of the labeled dataset is usually much smaller than that of the unlabeled dataset, i.e., \(n_l \ll n_u\). Denote \(\psi\) the feature generator, and \(h\) the task-specific head. The standard cross-entropy loss on weakly augmented labeled examples is \[L_\mathcal{L}(\psi, h)=\dfrac{1}{n_l} \sum_{i=1}^{n_l}L_{\text{CE}} \big((h \circ \psi \circ \alpha)(\mathbf{x}_i^l), {y}_i^l\big),\] where \(\alpha\) is the weak augmentation function. Since there are few labeled samples, the feature generator and the task-specific head will easily over-fit, and typical SSL methods use these pseudo labels on plenty of unlabeled data to decrease the generalization error. Different SSL methods design different pseudo labeling function \(\widehat{f}\) [2], [17], [45]. Take FixMatch [5] for an instance. FixMatch first generates predictions \(\widehat{\mathbf{p}} = (h\circ \psi \circ \alpha)(\mathbf{x})\) on a weakly augmented version of given unlabeled images, and adopts a confidence threshold \(\tau\) to filter out unreliable pseudo labels, \[\widehat{f}_{\psi, h}(\mathbf{x}) = \begin{cases} \arg \max \widehat{\mathbf{p}}, &\max \widehat{\mathbf{p}} \ge \tau, \\ -1, &\text{otherwise,} \end{cases}\] where \(\widehat{f}_{\psi, h}\) refers to the pseudo labeling by model \(h \circ \psi\), hyperparameter \(\tau\) specifies the threshold above which a pseudo label is retained and \(-1\) indicates that this pseudo label is ignored in training. Then FixMatch utilizes selected pseudo labels to train on strongly augmented unlabeled images, \[L_\mathcal{U}(\psi, h, \widehat{f}) = \dfrac{1}{n_u} \sum_{j=1}^{n_u} L_{\text{CE}} \big((h \circ \psi \circ \mathcal{A})(\mathbf{x}_j^u), \widehat{f}(\mathbf{x}_j^u)\big),\] where \(\widehat{f}\) is a notation of general pseudo labeling function and \(\mathcal{A}\) is the strong augmentation function. As shown in Figure 4 (a), the optimization objective for FixMatch is \[\label{eq:fixmatch95optimization} \min_{\psi, h} L_\mathcal{L}(\psi, h) + \lambda L_\mathcal{U}(\psi, h, \widehat{f}_{\psi, h}),\tag{1}\] where \(\lambda\) is the trade-off between the loss on labeled data and that on unlabeled data. FixMatch filters out low-confidence samples during the pseudo labeling process, yet two issues remain: (1) The pseudo labels are generated and utilized by the same head, which leads to the training bias, i.e., the errors of the model might be amplified as the self-training progresses. (2) When trained with extreme few labeled samples, the problem of unreliable pseudo labeling caused by data bias cannot be ignored anymore even with the confidence threshold mechanism. To tackle the above issues, we propose two important designs to decrease training bias and data bias in Section [sec:agent95head] and 4.2 respectively.

4.1 Generate and utilize pseudo labels independently↩︎

The training bias of FixMatch stems from the way of training on the pseudo labels generated by itself. To alleviate this bias, some methods turn to generate pseudo labels from a better teacher model, such as the moving average of the original model [3] in Figure 4 (b) or the model obtained from the previous round of training [24] in Figure 4 (c), and then utilize these pseudo labels to train both the feature generator \(\psi\) and the task-specific head \(h\). However, there is still a tight relationship between the teacher model that generates pseudo labels and the student model that utilizes pseudo labels in the above methods, and the decision hyperplanes of the student model \(h \circ \psi\) strongly depend on the biased pseudo labeling \(\widehat{f}\). As a result, training bias is still large in the self-training process.

To further decrease the training bias when utilizing the pseudo labels, we optimize the task-specific head \(h\), only with the clean labels on \(\mathcal{L}\) and without any unreliable pseudo labels from \(\mathcal{U}\). To prevent the deep models from over-fitting to the few labeled samples, we still use pseudo labels, but only for learning a better representation. As shown in Figure 4 (d), we introduce a pseudo head \(h_{\text{pseudo}}\), which is connected to the feature generator \(\psi\) and only optimized with pseudo labels from \(\mathcal{U}\). Then the training objective is \[\begin{align} \label{eq:agent} \min_{\psi, h, h_{\text{pseudo}}} L_\mathcal{L}(\psi, h) + \lambda L_\mathcal{U}(\psi, h_{\text{pseudo}}, \widehat{f}_{\psi, h}), \end{align}\tag{2}\] where the pseudo labels are generated by head \(h\) and utilized by a completely parameter independent pseudo head \(h_{\text{pseudo}}\). Although \(h\) and \(h_{\text{pseudo}}\) are fed with features from the same backbone network, their parameters are independent, thus training the pseudo head \(h_{\text{pseudo}}\) with some wrong pseudo labels will not accumulate the bias of head \(h\) directly in the iterative self-training process. Note that the pseudo head \(h_{\text{pseudo}}\) is only responsible for gradient backpropagation to the feature generator \(\psi\) during training and will be discarded during inference, and thus will introduce no inference cost.

4.2 Reduce generation of erroneous pseudo labels↩︎

Section [sec:agent95head] presents a solution to reduce the training bias, yet the data bias still exists in the pseudo labeling \(\widehat{f}\). As shown in Figure 5 (a), due to the data bias, labeled samples of each class have different distances to the decision hyperplanes in the representation space, which leads to a deviation between the learned hyperplanes and the real decision hyperplanes, especially when the size of labeled samples is very small. As a result, pseudo labeling \(\widehat{f}\) is very likely to generate incorrect pseudo labels on unlabeled data points that are close to these biased decision hyperplanes. And our objective now is to optimize the feature representations to reduce the data bias, and finally improve the quality of pseudo labels.

Since we have no labels for \(\mathcal{U}\), we cannot directly measure the data bias and thereby reduce it. Yet training bias has some correlations with data bias. Recall in Section [sec:agent95head], the task-specific head \(h\) is only optimized with clean labeled data, since optimization with incorrect pseudo labels will push the learned hyperplanes in a more biased direction and lead to the training bias. Therefore, training bias can be considered as the accumulation of data bias with inappropriate utilization of pseudo labels, which is training algorithm dependent. And the worst training bias that can be achieved by some self-training methods is a good measure of data bias. Specifically, the worst training bias corresponds to the worst possible head \(h'\) learned by pseudo labeling, such that \(h'\) predicts correctly on all the labeled samples \(\mathcal{L}\) while making as many mistakes as possible on unlabeled data \(\mathcal{U}\), \[\label{eq:worst95case} h_{\text{worst}}(\psi) = \arg\max_{h'} L_\mathcal{U}(\psi, h', \widehat{f}_{\psi, h}) - L_\mathcal{L}(\psi, h'),\tag{3}\] where the mistakes of \(h'\) on unlabeled data are estimated by its discrepancy with the current pseudo labeling function \(\widehat{f}\). Equation 3 aims to find the worst-case of task-specific head \(h\) that might be learned in the future when trained with pseudo labeling on the current feature generator \(\psi\) and the current data sampling. It is also the worst hyperplanes as shown in Figure 5 (b), which deviates as much as possible from the currently learned hyperplanes while ensuring that all labeled samples are correctly distinguished. Note that Equation 3 measures the degree of data bias, which depends on the feature representations generated by \(\psi\), thus we can adversarially optimize feature generator \(\psi\) to indirectly decrease the data bias, \[\label{eq:worst95case95objective95psi} \min_\psi L_\mathcal{U}(\psi, h_{\text{worst}}(\psi), \widehat{f}_{\psi, h}) - L_\mathcal{L}(\psi, h_{\text{worst}}(\psi)).\tag{4}\] As shown in Figure 5 (c), Equation 4 encourages the feature of unlabeled samples to be distinguished correctly even by the worst hyperplanes, i.e., be generated far away from the current hyperplanes, thereby reducing the data bias in feature representations.

4.2.0.1 Overall loss.

The final objective of the Debiased Self-Training (DST) approach is to reduce both training bias and data bias. The overall loss function simultaneously decouples the generation and utilization of pseudo-labels and avoids the worst-case hyperplanes. This is achieved by unifying Equations 2 –4 into a minimax game: \[\begin{align} \label{eq:overall} \min_{\psi, h, h_{\text{pseudo}}} \max_{h'} L_\mathcal{L}(\psi, h) + L_\mathcal{U}(\psi, h_{\text{pseudo}}, \widehat{f}_{\psi, h}) + \big( L_\mathcal{U}(\psi, h', \widehat{f}_{\psi, h}) - L_\mathcal{L}(\psi, h')\big). \end{align}\tag{5}\]

5 Experiments↩︎

Following [5], [17], we evaluate Debiased Self-Training (DST) with random initialization on common SSL datasets, including CIFAR-10 [46], CIFAR-100 [46], SVHN [47] and STL-10 [48]. Following [31], we also evaluate DST with both supervised pre-trained models and unsupervised pre-trained models on \(11\) downstream tasks, including (1) superordinate-level object classification: CIFAR-10 [46], CIFAR-100 [46], Caltech-101 [49]; (2) fine-grained object classification: Food-101 [50], CUB-200-2011 [51], Stanford Cars [52], FGVC Aircraft [53], OxfordIIIT Pets [54], Oxford Flowers [55]; (3) texture classification: DTD [56]; (4) scene classification: SUN397 [57]. The complete training dataset size ranges from \(2,040\) to \(75, 750\) and the number of classes ranges from \(10\) to \(397\). Following [58], we report mean accuracy per-class on Caltech-101, FGVC Aircraft, OxfordIIIT Pets, Oxford Flowers, and top-1 accuracy for other datasets. Following [5], we construct a labeled subset with \(4\) labels per category to verify the effectiveness of DST in extremely label-scarce settings. To make a fair comparison, we keep the labeled subset for each dataset the same throughout our experiments.

For experiments with random initialization, we follow [5] and adopt Wide ResNet variants [59]. For experiments with pre-trained models, we adopt ResNet50 [60] with an input size of \(224\times224\) and pre-trained on ImageNet [61]. We adopt MoCo v2 [62] as unsupervised pre-trained models. We compare our method with many state-of-the-art SSL methods, including Pseudo Label [2], \(\Pi\)-Model [23], Mean Teacher [3], VAT [39], ALI [37], RAT [42], UDA [16], MixMatch [14], ReMixMatch [15], FixMatch [5], Dash [17], Self-Tuning [31], FlexMatch [18] and DebiasMatch [13].

When training from scratch, we adopt the same hyperparameters as FixMatch [5], with learning rate of \(0.03\), mini-batch size of \(512\). For other experiments, we use SGD with momentum \(0.9\) and learning rates in \(\{0.001, 0.003, 0.01, 0.03\}\). The mini-batch size is set to \(64\) following [63]. For each image, we first apply random-resize-crop and then use RandAugment [64] for strong augmentation \(\mathcal{A}\) and random-horizontal-flip for weak augmentation \(\alpha\). More details on hyperparameter selection can be found in Appendix 7.2. Each experiment is repeated three times with different random seeds.

We have released a benchmark containing both the code for our method and that for all the baselines at https://github.com/thuml/Debiased-Self-Training.

5.1 Main results↩︎

Table 6 shows that DST yields consistent improvement on all tasks. On the challenging CIFAR-100 and STL-10 tasks, DST boosts the accuracy of FixMatch and FlexMatch by \(\boldsymbol{8.3\%}\) and \(\boldsymbol{10.7\%}\), respectively. Figure [fig:scratch95loss95and95acc] depicts the top-1 accuracy during the training procedure on CIFAR-100. We observe that the performance of FixMatch suffers from significant fluctuations during training. In contrast, the accuracy of DST (FixMatch) increases steadily and surpasses the best accuracy of FixMatch by \(\boldsymbol{10.9\%}\), relatively. Note that the accuracy of FlexMatch also drops by over \(6\%\) in the final stages of training while DST (FlexMatch) suffers from a much smaller drop by reducing erroneous pseudo labels during the self-training process. Besides, DST also improves the performance balance across categories (see Appendix 8.2).

5.2 Transfer from a pre-trained model↩︎

Supervisied pre-training. Table 1 reveals that typical self-training methods, e.g. FixMatch, lead to relatively mild improvements with supervised pre-trained models, which is consistent with previous findings [31], [63]. In contrast, incorporating DST into FixMatch significantly boosts the performance and surpasses FixMatch by \(\boldsymbol{19.9\%}\) on all datasets. With a pre-trained model, self-training has better training stability. Yet once the performance degradation occurs, the process is also irreversible (Appendix 8.1), partly due to the catastrophic forgetting of pre-trained representation. Also, self-training suffers from a more severe performance imbalance across classes (Appendix 8.2). DST effectively tackles these issues, indicating the importance of reducing bias.

Unsupervised pre-training. Table 1 shows that with unsupervised pre-trained models, more methods suffer from performance degradation after self-training on the unlabeled data. The difficulty comes from that the unsupervised pre-training task has a larger task discrepancy with the downstream classification tasks than the supervised pre-training task. Thus, the representations learned by unsupervised pre-trained models usually exhibit stronger data bias, and inappropriate usage of pseudo labels will lead to rapid accumulation errors and increase the training bias. By eliminating training bias and reducing data bias, DST brings improvement on all datasets and relatively outperforms FixMatch by \(\boldsymbol{23.5\%}\) on average, superior to FlexMatch and DebiasMatch in \(9\) and \(10\) tasks, respectively.

5.3 Ablation studies↩︎

We examine the design of our method on CIFAR-100 in Table 2 and have the following findings. (1) Compared with Mutual Learning [26], [65], where two heads provide pseudo labels to each other, the independent mechanism in our method where one head is only responsible for generating pseudo labels and the other head only uses them for self-training can better reduce the training bias. (2) A nonlinear pseudo head is always better than a linear pseudo head. We conjecture that nonlinear projection can reduce the degeneration of representation with biased pseudo labels. (3) The worst-case estimation of pseudo labeling improves the performance by large margins.

5.4 Analysis↩︎

To further investigate how DST improves pseudo labeling and self-training performance, we conduct some analysis on CIFAR-100. For simplicity, we only give the results with supervised pre-trained models. More comparisons can be found in Appendix 8.4.

DST improves both the quantity and quality of pseudo labels. As shown in Figures 7 (a) and 7 (b), FixMatch exploits unlabeled data aggressively, on average producing more than \(70\%\) pseudo labels during training. But the cost is that the accuracy of pseudo labels continues to drop, eventually falling below \(60\%\), which is consistent with our motivation in Section 3 that inappropriate utilization of pseudo labels will in turn enlarges the training bias. On the contrary, the accuracy of pseudo labels in DST suffers from a smaller drop. Rather, it keeps rising afterward and exceeds \(70\%\) throughout the training. Besides, DST generates more pseudo labels in the later stages of training.

DST generates better pseudo labels for poorly-behaved classes. To measure the quantity of pseudo labels on poorly-behaved classes, we calculate the class imbalance ratio \(I\) on a class-balanced validation set, \(I={{\max}_cN(c)}/{{\min}_{c'} N(c')}\), where \(N(c)\) denotes the number of predictions that fall into category \(c\). As shown in Figure 7 (c), the class imbalance ratio of FixMatch rises rapidly and reaches infinity after \(5000\) iterations, indicating that the model completely ignores those poorly-learned classes. To measure the quality of pseudo labels on poorly-behaved classes, we calculate the average accuracy of \(10\) or \(20\) worst-behaved classes in Figure 7 (d). The average accuracy on the worst \(20\) classes of FixMatch is only 1.0%. By reducing training bias with the pseudo head and data bias with the worst-case estimation, the average accuracy balloons to 28.5% and 34.5%, respectively.

5.5 Convergence and computation cost of the min-max optimization↩︎

We optimize \(\psi\) and \(h'\) with stochastic gradient descent alternatively. The optimization can be viewed as an alternative form of GAN [33]. Figure 8 shows that the worst-case error rate of \(h'\) and worst-case loss in Equation 4 first increase (\(h'\) dominates), and then gradually decrease and converge (\(\psi\) dominates). When training \(1000k\) iterations on CIFAR-100 using \(4\) 2080 Ti GPUs, FixMatch takes \(104\) hours while DST takes \(111\) hours, only a \(7\%\) increase in time. Note that DST introduces no additional computation cost during inference.

5.6 DST as a general add-on↩︎

We incorporate DST into several representative self-training methods, including FixMatch [5], Mean Teacher [3], Noisy Student [24], DivideMix [27] and FlexMatch [18]. Implementation details of DST versions of these methods can be found in Appendix [appendix:framework]. Table [table:framework] compares the original and DST versions of these methods on CIFAR-100 with both supervised pre-trained and unsupervised pre-trained models. Results show that the proposed DST yields large improvement on all these self-training methods, indicating that self-training bias widely exists in existing vanilla or sophisticated self-training methods and DST can serve as a universal add-on to reduce the bias.

6 Conclusion↩︎

To mitigate the requirement for labeled data, pseudo labels are widely used on the unlabeled data, yet they suffer from severe confirmation bias. In this paper, we systematically delved into the bias issues and present Debiased Self-Training (DST), a novel approach to decrease bias in self-training. Experimentally, DST achieves state-of-the-art performance on \(13\) semi-supervised learning tasks and can serve as a universal and beneficial add-on for existing self-training methods.

Acknowledgements↩︎

This work was supported by the National Key Research and Development Plan (2021YFB1715200), National Natural Science Foundation of China (62022050 and 62021002), Beijing Nova Program (Z201100006820041), BNRist Innovation Fund (BNR2021RC01002), and Kuaishou Research Fund.

7 Implementation Details↩︎

Our code is based on PyTorch [66]. The following are the implementation details of our experiments. We have released the code for our method and that for all the baselines at https://github.com/thuml/Debiased-Self-Training.

7.1 Architecture↩︎

The architectures of different classifier heads are as follows. For nonlinear heads, we adopt Dropout [41] to alleviate over-fitting.

• Linear main head: Linear-Softmax;

• Nonlinear pseudo head: Linear-ReLU-Dropout-Linear-Softmax;

• Worst-case head: Linear-ReLU-Dropout-Linear-Softmax.

7.2 Hyperparameters↩︎

For experiments without pre-trained models, we use Wide ResNet-28-2 [59] for CIFAR-10 and SVHN, WRN-28-8 for CIFAR-100, WRN-37-2 for STL-10 and adopt the same hyperparameters as FixMatch [5]. Specifically, we use learning rate of \(0.03\), mini-batch size of \(512\) (\(64\) for labeled data, \(448\) for unlabeled data), weight decay in \(\{0.0005, 0.001\}\), unlabeled loss weight \(\lambda=1.0\) and train for \(1000k\) iterations. For our method, we set the projection dimension of the pseudo head and worst-case head to \(2\times n_{\text{embedding}}\), where \(n_{\text{embedding}}\) denotes the dimension of backbone network output.

For experiments with pre-trained models, we use SGD with momentum of \(0.9\). We choose weight decay in \(\{0.0005, 0.001\}\), learning rates in \(\{0.001, 0.003, 0.01, 0.03\}\). We train for \(40k\) iterations and use the cosine learning rate schedule. The mini-batch size is set to \(64\). Besides, we tune the following algorithm-specific hyperparameters.

\(\Pi\)-Model. We search unlabeled loss weight \(\lambda\) in \(\{0.1, 0.3, 1.0, 3.0\}\), warm-up iterations of unlabeled loss in \(\{5\times10^3, 10^4\}\).

Mean Teacher. We fix the exponential moving average hyperparameter \(\alpha\) to \(0.999\). We search unlabeled loss weight \(\lambda\) in \(\{0.1, 0.3, 1.0, 3.0\}\), warm-up iterations of unlabeled loss in \(\{5\times10^3, 10^4\}\).

Pseudo Label. We search confidence threshold \(\tau\) in \(\{0.7, 0.8, 0.9, 0.95\}\), unlabeled loss weight \(\lambda\) in \(\{0.1, 0.3, 1.0, 3.0\}\).

FixMatch. The same as Pseudo Label.

UDA. We search temperature in \(\{0.1, 0.5, 1.0, 2.0\}\), unlabeled loss weight \(\lambda\) in \(\{0.1, 0.3, 1.0, 3.0\}\).

Noisy Student. We search temperature in \(\{0.1, 0.5, 1.0, 2.0\}\), unlabeled loss weight \(\lambda\) in \(\{0.1, 0.3, 1.0, 3.0\}\). We iterate \(3\) rounds, excluding the round that trains with only labeled data. The final performance is reported.

Self-Tuning. We try queue size in \(\{24, 32\}\) and use temperature \(0.07\), projection dimension \(1024\), same as the original paper [31].

FlexMatch. The same as Pseudo Label. Besides, we find it better to turn off threshold warm-up when using pre-trained models.

DebiasMatch. We search confidence threshold \(\tau\) in \(\{0.7, 0.8, 0.9, 0.95\}\), unlabeled loss weight \(\lambda\) in \(\{0.1, 0.3, 1.0, 3.0\}\), debias factor in \(\{0.1, 0.3, 1.0, 3.0\}\) and fix momentum to \(0.999\). For a fair comparison, we do not exploit additional supervision from pre-trained CLIP models [67].

DST. We set the confidence threshold to \(0.7\) by default. We fix the projection dimension of the pseudo head and the worst-case head to \(2048\). The trade-off hyperparameter \(\lambda\) is set to \(1\).

7.3 DST as a general add-on to previous self-training methods↩︎

In this section, we will illustrate how to incorporate DST into \(4\) typical self-training methods, including FixMatch, FlexMatch, Mean Teacher, and Noisy Student. We will mainly focus on the modification to the generation and utilization of pseudo labels, and omit the introduction of the worst-case heads since they are the same across different self-training methods.

Debiased FixMatch. As shown in Figure 9 (a), the pseudo labels on the unlabeled data are generated by the main head \(h\) and utilized by the pseudo head \(h_{\text{pseudo}}\). The main head \(h\) is only trained on the clean labeled data.

Debiased FlexMatch. The same as Debiased FixMatch. The learning status of each category is estimated by the main head \(h\).

Debiased Mean Teacher. As shown in Figure 9 (b), the pseudo labels on the unlabeled data are generated by the exponential moving average of the main head \(h\) and utilized by the pseudo head \(h_{\text{pseudo}}\). The main head \(h\) is only trained on the clean labeled data.

Debiased Noisy Student. As shown in Figure 9 (c), the pseudo labels on the unlabeled data are generated by the head \(h\) of previous round \(T-1\) and utilized by the pseudo head \(h_{\text{pseudo}}\). The main head \(h\) is only trained on the clean labeled data.

8 More Experimental Results↩︎

8.1 Experiments on training stability↩︎

We further explore the training stability of FixMatch when using pre-trained models on various tasks. Figure 10 illustrates several failure cases of FixMatch.

Figures 10 (a) and 10 (b) show {when the performance of the pre-trained models declines, it cannot be recovered later

. Note that we try confidence threshold in \(\{0.7, 0.8, 0.9, 0.95\}\) and get similar results.

Figure 10 (c) demonstrates a complete failure case of FixMatch. With unsupervised pre-trained models and a confidence threshold of \(0.7\), there can be a lot of noise in pseudo labels and thus the performance of FixMatch declines severely. Note this result is not the entry we report in Table 1 (threshold is set to \(0.9\) for this dataset). Instead, we aim to show that DST improves the training stability when there is much noise.

8.2 Experiments on performance balance between categories↩︎

Figures 11 and 12 plot the top-1 accuracy of each category on CIFAR-100 yielded by self-training on \(400\) labels and unlabeled data when using supervised pre-trained models or training from scratch, respectively. The results are consistent with our previous analysis (Section 5.4) that DST improves the performance of those poorly-behaved categories.

8.3 Experiments with varying amounts of labeled data↩︎

Table 3 reports the performance of DST with \(1000\) labels on CIFAR-100 with different pre-trained models. DST outperforms FlexMatch and DebiasMatch under both settings.

Figure 13 plots the accuracy of FixMatch and DST when the number of labeled samples per class varies from \(1\) to \(25\) on CIFAR-100 with supervised pre-trained models. Experiments show that DST is less sensitive to the amount of labeled data than FixMatch and yield consistent improvement.

8.4 Analysis on the behavior of pseudo labels↩︎

8.4.1 Using unsupervised pre-trained models↩︎

In this subsection, we explore the behavior of pseudo labels with unsupervised pre-trained models. Concretely, we focus on the quantity, accuracy as well as class imbalance ratio \(I\) of pseudo labels. Recall that \(I={{\max}_cN(c)}/{{\min}_{c'} N(c')}\), where \(N(c)\) denotes the number of predictions that fall into category \(c\). Figure 14 shows the results on CIFAR-100 with \(400\) labels. We observe the same phenomenon that DST effectively reduces the bias of pseudo labels, thereby improving the self-training process.

8.4.2 Comparison with other methods↩︎

We consider two lines of work that promote the quality of pseudo labels by (1) using dynamic threshold, including Dash [17] and FlexMatch [18]; (2) adopting multi-view training, including Co-training [25] and Multi-task Tri-training [28]. Table 4 shows that DST outperforms baselines by considerable margins. Figure 15 plots the quality of pseudo labels and reveals that our method can better debias pseudo labeling and improve the quality of pseudo labels.

8.5 Ablation study on nonlinear main classifier head↩︎

Experiments suggest that using a nonlinear pseudo head improves performance. We further explore how things are going for the main head. As shown in Table 5, using nonlinear main head or not results in a similar performance on average. We conjecture this is because a nonlinear main head is more likely to over-fit with few labeled samples.

9 Broader Impact↩︎

First, our research helps improve the performance and training stability of various existing self-training methods, especially when the labeled data is scarce. Second, our proposed method is simple yet effective and thus can potentially reduce the labeling cost of many real-world machine learning applications. Finally, our research helps reduce the bias of self-training models and improves their performance balance.

References↩︎

---

1. Equal contribution.↩︎