1 Introduction↩︎

With the rapid advancement of image generation technology, the number of AI-generated images (AIGIs) has surged, and they are now widely used across various fields. However, the quality of these AIGIs varies greatly, and low-quality AIGIs severely impair user experience. Therefore, efficiently filtering high-quality images from large-scale AIGIs is crucial.

To address this, the task of AI-generated image quality assessment (AIGIQA) has gained increasing attention. Most existing methods [1]–[11] process image inputs in two primary ways: 1) global processing, which extracts features directly from the whole image—computationally efficient but prone to loss of fine-grained details; and 2) local–global hybrid processing, which aggregates features from multiple local patches and the global image—more accurate but computationally expensive and unsuitable for large-scale real-time assessment. This accuracy–efficiency trade-off becomes particularly pronounced for high-resolution images. Thus, we raise the key research question: Is there an innovative approach that can effectively explore and integrate multi-level visual representations (from local details to global semantics) while maintaining high computational efficiency?

To tackle this challenge, we propose Patch Knowledge Transfer (PKT), a knowledge distillation-based framework that unifies comprehensive visual processing with efficient inference. As shown in Fig. 1 (a), we design a dual-branch architecture: the teacher model employs local–global hybrid processing to build comprehensive quality priors through multi-scale feature extraction; the student model retains only the global processing branch but learns the teacher’s multi-level representational capability via feature distillation. This allows the student model to maintain efficient single-image forward computation while inheriting the teacher’s rich quality discrimination knowledge, ultimately achieving an optimal balance between accuracy and efficiency (Fig. 1).

To ensure the student fully assimilates the teacher’s representations, we design a multi-level knowledge transfer scheme with two core supervision mechanisms: feature-level knowledge transfer, which aligns features at the last encoder layer via a feature alignment loss, and output-level knowledge distillation, which minimizes the KL divergence between the two models’ prediction distributions. This hierarchical strategy transfers not only explicit discriminative knowledge but also the teacher’s internal feature extraction paradigm.

Extensive experiments on four mainstream AIGIQA databases demonstrate that PKT achieves a better trade-off between efficiency and performance, including AGIQA-1K [12], AGIQA-3K [13], AIGCIQA2023 [14], and PKU-AIGIQA-4K [15]. On AGIQA-1K, compared to the local–global hybrid teacher model, PKT reduces computational cost (FLOPs) by 67.7% while maintaining comparable performance; compared to the global-only baseline, it improves SRCC and PLCC by 0.0098 and 0.0091, respectively, under the same computational budget. Notably, the student model even surpasses the teacher on certain metrics, indicating effective knowledge transfer and performance enhancement.

2 Related Work↩︎

Recent years have witnessed remarkable advancements in the field of AIGIQA. Qu et al. [3] develop IP-IQA, a CLIP-based framework that jointly processes AIGIs and their text prompts. Yuan et al. [5] introduce PSCR, which allows for inter-comparison among images in the database. Zhou et al. [9] introduce AMFF-Net, which employs multi-scale image inputs to extract features and predict multiple quality dimensions. Peng et al. [6] propose ICPE, an enhanced version of LIQE that incorporates text-image correlation analysis. Yu et al. [7] present SF-IQA, a unique metric combining quality and similarity assessments through score fusion. Yang et al. [8] present MoE-AGIQA, a hybrid expert system integrating visual and semantic perception. Tang et al. [16] present CLIP-AGIQA, leveraging CLIP’s multimodal knowledge for quality regression. These methods typically employ sophisticated feature extraction strategies to explore and integrate multi-level visual information (spanning local details to global semantics). While such strategies can enhance model performance, they substantially increase computational complexity, making it difficult to meet the real-time inference requirements. In contrast, we propose a knowledge distillation-based optimization framework PKT in this paper, which achieves efficient inference while ensuring effective processing of multi-level visual information.

3 Approach↩︎

In this section, we first introduce the fundamental concepts and problem formulation. Subsequently, we provide a comprehensive elaboration on the operational mechanism of the proposed Patch Knowledge Transfer (PKT) approach. Fig. 2 illustrates the overall framework of PKT.

3.1 Pipeline Overview↩︎

In recent years, the application of the CLIP model in image quality assessment (IQA) has attracted widespread attention [6], [9], [11], [16]–[18]. In this work, both the teacher and student models largely follow the architecture of LIQE. LIQE [18] computes cosine similarity between the image and a structured textual description: “a photo with {q} quality”, where q corresponds to one of five quality levels (bad, poor, fair, good, perfect) associated with five score values \(V=\{v_i\}_{i=1}^5\). The similarities are transformed into probabilities via a Softmax function, and a weighted sum is applied to produce the final image quality score.

Given an image \(X\), the teacher model samples \(N\) image patches \(X_p=\{x^i_p\}_{i=1}^N\) from the input image and inputs them along with the scaled image \(X_r\) to the encoder, while the student model processes only the scaled image. This process can be represented as: \[\begin{align} F_t = E^I_t(X_p, X_r), \tag{1} \\ F_s = E^I_s(X_r), \tag{2} \end{align}\] Here, \(F_t=\{F^i_t\}_{i=1}^{N+1}\) and \(F_s\) represent the multi-patch image features extracted by the teacher model and the single-scale image features extracted by student model, respectively. \(E^I_t\) and \(E^I_s\) denote the image encoders of the teacher and student models, respectively. For the designed text template, the teacher and student models use their respective text encoders to extract text features \(F^T_t\) and \(F^T_s\). Subsequently, we calculate the cosine similarity between the image features and the text features as follows: \[\begin{align} S_t = \frac{F^T_t \cdot F_t}{\lVert F^T_t \rVert \lVert F_t \rVert}, \tag{3} \\ S_s = \frac{F^T_s \cdot F_s}{\lVert F^T_s \rVert \lVert F_s \rVert}, \tag{4} \end{align}\] Here, \(S_t=\{S^i_t\}_{i=1}^{N+1}\) and \(S_s\) represent the cosine similarity calculated by the teacher model and the student model, respectively. We then use a Softmax function to transform the cosine similarities \(S_t=\{S^i_t\}_{i=1}^{N+1}\) and \(S_s\) into probabilities \(\mathbb{P}_t=\{\mathbb{P}^i_t\}_{i=1}^{N+1}\) and \(P_s=\{p^i_s\}_{i=1}^{5}\) for the five quality levels. The output probability distribution \(P_t=\{p^i_t\}_{i=1}^{5}\) of the teacher model is obtained by averaging the predicted probabilities on five quality levels for all image patches and the scaled image, which can be expressed as: \[\begin{align} P_t = \frac{1}{N+1} \sum_{i=1}^{N+1} \mathbb{P}^i_t, \label{Eq5} \end{align}\tag{5}\] Finally, we calculate the final predicted score \(\hat{y_t}\) and \(\hat{y_s}\) of the teacher model and the student model through a weighted summation, which can be represented as: \[\begin{align} \hat{y}_t = \sum_{i=1}^{5} p^i_t \times v_i, i=1, \cdots, 5 , \tag{6} \\ \hat{y}_s = \sum_{i=1}^{5} p^i_s \times v_i, i=1, \cdots, 5 , \tag{7} \end{align}\]

3.2 Patch Knowledge Transfer↩︎

This paper introduces Patch Knowledge Transfer, whose key innovation lies in the design of a knowledge distillation mechanism that enables the student model to inherit visual priors extracted from multi-region features by the teacher model, thereby achieving a balance between accuracy and efficiency. The details are as follows.

Training Loss. The training of the teacher model is achieved by minimizing the Mean Squared Error (MSE) loss between its predicted scores \(\hat{y_t}\) and the true quality scores \(y\): \[\begin{align} \mathcal{L}_{\text{teacher}} = \mathcal{L}_{\text{mse}}(\hat{y_t}, y ) = \| \hat{y_t} - y \|_2^2 ,\label{Eq8} \end{align}\tag{8}\] In order for the student model to effectively learn from the teacher model’s knowledge, we introduce multi-level supervision during the training process: 1) Feature-level knowledge transfer: By constraining the similarity between last layer features of the student model and the corresponding features of the teacher model; 2) Output-level knowledge distillation: Ensuring that the output probability distribution of the student model approaches the predicted distribution of the teacher model. To align the dimensions between the student and teacher models during feature distillation, we applied a simple average aggregation to multiple features output by the teacher model. Specifically, the overall loss function of the student model can be expressed as: \[\begin{align} \mathcal{L}_{\text{student}} &= \mathcal{L}_{\text{dist}}(F_s, F_t) + \mathcal{L}_{\text{kl}}(P_s,P_t) + \mathcal{L}_{\text{mse}}(\hat{y_s}, y) \nonumber \\ &= (1 - \frac{F_s \cdot \frac{1}{N+1} \sum_{i=1}^{N+1} F^i_t }{\lVert F_s \rVert \lVert \frac{1}{N+1} \sum_{i=1}^{N+1} F^i_t \rVert}) + \sum P_t \text{log} \frac{P_t}{P_s} \nonumber \\ &+ \| \hat{y_s} - y \|_2^2, \label{Eq9} \end{align}\tag{9}\] Where \(\mathcal{L}_{\text{dist}}\) is the feature-level similarity loss between the student and teacher models’ last layer outputs, \(\mathcal{L}_{\text{kl}}\) is the output-level distillation loss between the predicted output probability distributions of the student and teacher models, \(\mathcal{L}_{\text{mse}}\) is the mean squared error loss between the predicted quality score and the true score.

One-Stage Training and Two-Stage Training. This study explores two knowledge distillation training paradigms: 1) Joint training mode (PKT-1), where the teacher and student models are trained end-to-end with synchronized optimization of parameters, with the objective function being: \[\begin{align} \mathcal{L}_{\text{total}} &= \mathcal{L}_{\text{dist}}(F_s, F_t) + \mathcal{L}_{\text{kl}}(P_s, P_t) + \mathcal{L}_{\text{mse}}(\hat{y_s}, y) \nonumber \\ &+ \mathcal{L}_{\text{mse}}(\hat{y_t}, y ); \label{Eq10} \end{align}\tag{10}\] 2) Stage-wise training mode (PKT-2), where the teacher model is first trained independently, and its parameters are then fixed to guide the training of the student model.

4 Experiment↩︎

4.1 Databases and Evaluation Criteria↩︎

We evaluate our approach on 4 AIGIQA benchmarks, including AGIQA-1K [12], AGIQA-3K [13], AIGCIQA2023 [14], and PKU-AIGIQA-4K [15]. This study employs two widely adopted image quality assessment (IQA) metrics: the Spearman Rank Correlation Coefficient (SRCC) to evaluate prediction monotonicity and the Pearson Linear Correlation Coefficient (PLCC) to assess prediction accuracy.

4.2 Implementation Details↩︎

Our experiments are conducted on the NVIDIA A100, using PyTorch 1.11.0 and CUDA 11.3 for both training and testing. The batch size \(B\) is set to \(8\) for training and set to \(20\) for testing. We utilize the Adam optimizer [19] with a learning rate of \(5 \times 10^{-6}\) and weight decay of \(1 \times 10^{-3}\). Unless otherwise specified, the default configurations in this study are as follows: the image encoder employs ViT-B/32 [20], the teacher model adopts an overlapping patch sampling strategy with 9 sampled patches, and a two-stage training strategy is adopted by default during model training. Additionally, data augmentation techniques including random flipping are implemented during training. For dataset splitting, we randomly split the dataset into training set and test set at a ratio of 4:1.

4.3 Results and Analysis↩︎

Comparisons with SOTA IQA Methods. We conduct extensive comparisons between our method and 9 SOTA IQA models: MUSIQ [1], HyperIQA [21], StairIQA [22], MANIQA [2], LIQE [18], Re-IQA [23], PSCR[5], AMFF-Net [9], and CLIP-AGIQA [16]. The experimental results in TABLE 1 demonstrate the relative performance across 4 AIGIQA databases, including AGIQA-1K, AGIQA-3K, AIGCIQA2023, and PKU-AIGIQA-4K. Experimental results demonstrate that our proposed PKT framework achieves a superior balance between model performance and computational efficiency compared to existing methods, as evidenced by two key aspects:

(1) Significant Performance Gains at Comparable Computational Cost: When compared to methods with similar FLOPs, PKT shows clear performance advantages. For example, on the AGIQA-1K benchmark:

• PKT-2 vs. HyperIQA: +0.0473 (SRCC) / +0.0275 (PLCC)

• PKT-2 vs. StairIQA: +0.0367 (SRCC) / +0.0288 (PLCC)

(2) Drastic FLOPs Reduction While Maintaining Competitive Performance: Compared to high-performance large models, PKT achieves comparable or better accuracy with significantly lower computational cost. For example, on the AGIQA-1K benchmark:

• PKT-2 vs. MANIQA: -95.98G FLOPs , +0.0203 (SRCC) / +0.0103 (PLCC)

• PKT-2 vs. MUSIQ: -113.88G FLOPs , +0.0501 (SRCC) / +0.0337 (PLCC)

• PKT-2 vs. LIQE: -41.25G FLOPs , +0.008 (SRCC) / +0.007 (PLCC)

Notably, our PKT-2 achieves SOTA average performance across 4 benchmark datasets. On the AGIQA-1K and AGIQA-3K databases, PKT-2 exhibits particularly dominant advantages, outperforming all competing methods comprehensively.

Comparisons with Multiple Patch-based Teacher and Baseline. To further validate the effectiveness of our proposed method, we systematically compared it with the teacher model using local-global hybrid processing and the student model without PKT (Baseline). The experiments were comprehensively verified using multiple image encoders, including ViT-B/32, ViT-B/16 and ViT-L/14 [20]. As shown in TABLE 2, the teacher model demonstrates superior performance due to its higher computational complexity (e.g., the ViT-L/14 teacher model has FLOPs as high as 540.75G). By introducing the PKT method, the student model not only significantly surpasses the baseline under the same computational cost but also approaches or even exceeds the teacher model’s performance on multiple metrics (e.g., evaluation results of ViT-B/16 and ViT-L/14), fully demonstrating effectiveness of PKT in knowledge transfer. For example, on the PKU-AIGIQA-4K benchmark, when using ViT-B/32 as the image encoder, our student model achieves performance improvements of 0.0178 and 0.0147 in SRCC and PLCC metrics, respectively, while maintaining comparable computational costs to the baseline. More notably, the proposed method reduces computational costs by 67.7% while achieving performance comparable to the teacher model, highlighting its advantage in balancing model efficiency and performance.

Study on feature alignment strategies. To thoroughly optimize the effectiveness of feature-level knowledge distillation, we conduct a systematic comparative study on different feature alignment strategies. The experimental results are presented in TABLE 3. Our findings demonstrate that, compared to Manhattan and Euclidean Distance minimization approaches, the cosine similarity maximization strategy exhibits significant advantages. This observation can be explained from the perspective of feature space geometry: cosine similarity focuses on directional consistency of feature vectors rather than absolute value matching, allowing the student model to learn the teacher’s feature distribution patterns more flexibly without being constrained by feature scale differences.

Ablation Study. In this section, we systematically investigate the impact of different loss function combinations on model performance, comparing seven distinct configurations. The baseline model employs only \(\mathcal{L}_{\text{mse}}\) without any teacher knowledge. Ablation studies on loss functions presented in TABLE 4 reveal three key findings: First, while \(\mathcal{L}_{\text{dist}}\) alone underperforms the baseline, its combination with \(\mathcal{L}_{\text{mse}}\) improves SRCC and PLCC by 0.0115 and 0.0089 respectively compared to baseline, demonstrating that feature-level transfer requires task-specific constraints for effective performance gain. Second, \(\mathcal{L}_{\text{kl}}\) alone surpasses the baseline, confirming the direct efficacy of output-level distillation. Third, the combined use of all three losses achieves optimal performance, validating our multi-level distillation strategy - feature-level and output-level knowledge transfer exhibit complementary effects, whose synergy maximizes distillation benefits.

4.4 Visualization and Analysis↩︎

To further demonstrate the effectiveness of our method, we first visualize the score prediction results of the baseline model and our proposed PKT method on the test set of the PKU-AIGIQA-4K database using scatter plots, as shown in Fig. 3. We observed that, compared to the baseline model, the predicted points of the proposed method are more densely distributed near the diagonal, which intuitively reflects that the student model using the PKT method achieves higher prediction accuracy. Additionally, we provide a comparison of the inference time between the student and teacher models trained using the proposed method, as shown in TABLE 5. Taking the teacher model as the baseline, the student model requires, on average, only 36% of the inference time for evaluating the same number of images. This means that, while achieving comparable or even superior performance to the teacher model, the student model improves inference efficiency by 64%. These results further validate the effectiveness of our method.

5 Conclusion↩︎

This paper presents PKT, a knowledge distillation-based optimization framework that addresses the critical challenge of balancing efficiency and accuracy in AIGIQA task. The proposed framework employs a teacher-student architecture with multi-level supervision during training to achieve efficient knowledge transfer. Extensive experiments on 4 AIGIQA benchmarks demonstrate that PKT outperforms existing IQA methods, maintaining high accuracy while significantly improving inference efficiency - particularly suitable for deployment scenarios with stringent real-time requirements. We expect the introduction of PKT will provide new insights and solutions for efficient quality assessment of AIGIs.

5.1 Similarity-based Region-Weighted Adaptive Feature Distillation↩︎

The loss function used for feature-level knowledge transfer mentioned above merely guides the feature learning of the student model by simply averaging multiple regional features of the teacher model, which may exhibit certain limitations. We argue that during feature transfer, different regional features should contribute differentially. To achieve more refined feature-level distillation, we propose Similarity-based Region-Weighted Adaptive Feature Distillation (SRWAFD). It introduces no additional learnable parameters. Instead, it directly utilizes the similarity between patch-wise features from the teacher model and the student features as weighting coefficients to differentiate the importance of distinct regional features, which dynamically evaluates the importance of each region in the teacher’s features for the student’s feature learning, thereby enabling more efficient feature knowledge distillation. The implementation involves two key steps: first, computing the cosine similarity \(C=\{c_i\}_{i=1}^{N+1}\) between each regional feature of the teacher and the student’s features as follws: \[\begin{align} c_i = \frac{F_s \cdot F^i_t}{\lVert F_s \rVert \lVert F^i_t \rVert} , i=1, \cdots,N+1; \label{Eq4616} \end{align}\tag{11}\] second, converting the cosine similarity into normalized attention weights \(W=\{w_i\}_{i=1}^{N+1}\) via the Softmax function: \[\begin{align} w_i = \frac{e^{c_i}}{\sum_{j=1}^{N+1}e^{c_j}}, i=1, \cdots, N+1 \label{Eq4617} \end{align}\tag{12}\] Ultimately, the loss function can be expressed as: \[\begin{align} \mathcal{L}_{\text{dist}}(F_s, F_t) = 1 - \frac{F_s \cdot \sum_{i=1}^{N+1} w_i F^i_t }{\lVert F_s \rVert \lVert \sum_{i=1}^{N+1} w_i F^i_t \rVert} \label{Eq11} \end{align}\tag{13}\]

5.2 Additional Experimental Results↩︎

In this section, we present additional experimental results that were not included in the main text.

Results on AIGIQA-20K. TABLE 1 compares the performance of multiple patch-based teacher models against baseline methods on the AIGIQA20K[24] dataset.

Study on Different Patches Sampling method. In this work, the teacher model adopts a local-global hybrid input approach that combines image patches with a downscaled full image for feature extraction. To thoroughly investigate the impact of different image patches sampling strategies in the teacher model on the performance of the student model, we systematically compare three typical sampling methods: random patches sampling (RPS), non-overlapping patches sampling (NOPS), and overlapping patches sampling (OPS). Experimental results presented in TABLE 2 demonstrate that the OPS strategy exhibits the best performance stability, delivering significant and consistent improvements across all tested datasets. In contrast, while RPS may achieve optimal results on certain datasets, it suffers from high performance variability and lacks stability. NOPS performs slightly worse than OPS across all three tested databases. Based on this analysis, we ultimately select OPS as the default patches sampling strategy, ensuring robustness and providing more reliable supervision signals for knowledge transfer to the student model.

Study on different number of sampled patches. To further optimize guidance effectiveness of the teacher model, we conduct an in-depth study on how the number of sampled image patches affects the performance of the student model. Based on the OPS strategy, we design comparative experiments to evaluate model performance with 9, 16, and 25 sampled patches. As shown in TABLE 8, the results indicate that the relationship between patch count and model performance is not strictly positive. When sampling 9 patches, the student model achieves the best average performance across all evaluation metrics. This finding suggests that a moderate amount of local information provides the most discriminative supervision signals for knowledge transfer, whereas excessive patches may introduce feature redundancy, ultimately impairing generalization. Additionally, computational cost increases linearly with patch count, yet performance gains remain marginal. Based on this analysis, we set the default number of sampled patches to 9 when employing the OPS strategy. Additionally, TABLE 3 analyzes how the teacher model’s performance varies with an increasing number of sampled patches.

Comparison of homogeneous self-distillation and heterogeneous distillation. To further investigate the impact of model architecture in knowledge distillation, we systematically compared two paradigms: homogeneous self-distillation (ViT-B/32 \(\rightarrow\) ViT-B/32) and heterogeneous distillation (ViT-B/16 \(\rightarrow\) ViT-B/32, ViT-L/14 \(\rightarrow\) ViT-B/32). When performing heterogeneous distillation where the student and teacher dimensions do not match—for instance, when distilling from ViT-L/14 to ViT-B/32, with student features of shape \((B, 512)\) and teacher features of \((B, N, 768)\)—we directly apply linear interpolation to the student’s features to adjust them to \((B, 768)\) for dimension alignment. Notably, this process does not introduce any additional parameters or mapping heads. As shown in TABLE 10, the experimental results reveal that although ViT-B/16 and ViT-L/14 teacher models outperform ViT-B/32 individually (shown in Table 2 of the mian text), their knowledge transfer efficacy is inferior to homogeneous self-distillation—achieving lower scores on most evaluation metrics. Notably, these larger teacher models also incur substantially higher training costs. Based on this empirical analysis, we ultimately select the homogeneous self-distillation scheme as our knowledge transfer mechanism, due to its more balanced computational efficiency and performance.

Additional experimental results about study on feature aggregation strategies. We conduct experiments to compare the effects of different feature aggregation strategies on student model performance, including: (1) simple average; (2) learned weights, which assigns weights to different regions via a fully connected layer and a sigmoid activation function; and (3) the proposed Similarity-based Region-Weighted Adaptive Feature Distillation (SRWAFD). As shown in TABLE 12, the results indicate that learned weights underperforms even simple average, whereas the proposed SRWAFD method further improves model performance without introducing any additional parameters, demonstrating its superiority.

Additional experimental results about comparisons under different temperature settings. Temperature \(T\) is a key parameter in knowledge distillation, designed to soften probability distributions during training. We further investigated the impact of different temperature coefficients on student model performance. Experimental results in TABLE 13 indicate that distillation achieves the best performance when no temperature scaling is applied (i.e., \(T\)=1) on the AGIQA-1K and AGIQA-3K dataset. Nevertheless, tuning \(T\) may yield benefits, as shown in the supplementary appendix. In this work, we set \(T\)=1 by default without employing temperature scaling to soften the outputs of the student and teacher models. This is because temperature softening does not always lead to a performance gain.

Impact of different loss weights. In the main text, the loss function for the students of PKT-2 is as follows: \[\begin{align} \mathcal{L}_{\text{student}} = \mathcal{L}_{\text{dist}}(F_s, F_t) + \mathcal{L}_{\text{kl}}(P_s,P_t) + \mathcal{L}_{\text{mse}}(\hat{y_s}, y), \label{qvobnuzp} \end{align}\tag{14}\] In the above formulation, the coefficients for different weights are set to 1. Here, we investigate the impact of assigning different weights to distinct loss functions. The new loss function is defined as follows: \[\begin{align} \mathcal{L}_{\text{student}} = \alpha \cdot \mathcal{L}_{\text{dist}}(F_s, F_t) + \beta \cdot \mathcal{L}_{\text{kl}}(P_s,P_t) + \gamma \cdot \mathcal{L}_{\text{mse}}(\hat{y_s}, y), \label{yfnsjolx} \end{align}\tag{15}\] Experimental results in Table 14 show that different loss weights have minimal impact on model performance.

Additional experimental results about robustness experiments. To validate the robustness of the proposed method, we conducted experiments under four different dataset splits using random seeds 0, 1, 2, and 42. The complete experimental results are presented in TABLE 4, TABLE 5, TABLE 17, TABLE 18, TABLE 19, TABLE 20, and TABLE 21. Additionally, we conduct cross-dataset evaluations as shown in TABLE 11. These results further demonstrate the robustness of the proposed method

5.3 Visualization and Analysis↩︎

This section presents additional visualization results to further demonstrate our method’s effectiveness.

Fig. 4 visualizes the score prediction scatter plots comparing the Baseline and PKT methods on the AGIQA-3K database. The denser distribution of data points along the diagonal in our PKT method demonstrates its enhanced prediction accuracy compared to the baseline approach. This visual evidence confirms that our knowledge transfer framework effectively improves score prediction reliability.

5.4 Discussion↩︎

A short discussion about situations where the student model occasionally outperforms the teacher model. We believe this is because the student model does not merely fit the labels of the dataset but learns from the richer signals of the teacher model, which encourages the formation of smoother and more generalized feature representations. It helps avoid overfitting to the details that the teacher model might have already overfitted.

References↩︎