1 Introduction↩︎

Identifying and supporting knowledge construction in online discussions remains a significant challenge in education [1]. As digital learning environments such as massive open online courses, discussion forums, and social media continue to expand, learners are producing vast amounts of textual interaction data [2], [3]. Understanding how knowledge is collaboratively constructed within these environments offers valuable insights into the processes of learning [4], [5].

Traditional analyses of knowledge construction have relied on manual content analysis. For instance, [6] proposed a five-dimensional framework for computer-mediated communication that includes participation, interaction, social, cognitive, and metacognitive dimensions, providing a systematic basis for interpreting learning processes in message transcripts. [4] introduced the Interaction Analysis Model (IAM), which conceptualizes the process of knowledge construction in computer mediated communication as a progressive sequence of phases, from sharing and comparing information to negotiating meaning, constructing new knowledge, and reaching mutual understanding. The IAM has since become a foundational framework for examining collaborative learning and knowledge building in online environments [7]. Subsequent studies further refined the analysis of knowledge construction. [8] employed mixed methods to investigate how groups build shared understanding, demonstrating that both detailed conversational exchanges and broader thematic patterns characterize computer supported collaborative learning. [9] proposed a multidimensional framework for argumentative knowledge construction, distinguishing participation, epistemic, argumentative, and social dimensions to capture how learners co-construct knowledge through discourse. Together, these studies advanced methodological approaches for examining knowledge construction in digital learning contexts.

Despite recent progress, manual coding remains labour intensive, time consuming, and susceptible to subjectivity [10]. Studies must establish and maintain interrater reliability, and disagreements often require negotiated coding [11]. To reduce delay and improve scalability, researchers have explored automated discourse classification for timely feedback. Deep learning and other supervised approaches have been used to identify phases of knowledge construction [12]. However, performance varies with language, context, and the availability of labelled data, and key challenges persist [13]. Recent advances in large pretrained language models have transformed natural language processing and offer improved contextual understanding for educational discourse analysis [1]. Models such as BERT, RoBERTa, and DeBERTa show strong performance on downstream tasks including stance detection, sentiment analysis, and argumentative discourse modelling [14], [15]. Their application to fine grained classification of knowledge construction, however, remains underexplored [3].

This study addresses that gap by proposing a DeBERTa-v3-large–based classifier that integrates focal loss, label smoothing, and R-Drop regularisation [16]–[18]. The approach aims to identify phases of knowledge construction in online discussions and to improve robustness under limited and imbalanced labelled data.

1.1 Knowledge Construction in Online Learning↩︎

Knowledge construction is a complex cognitive process that entails learners’ careful processing of information [19]. In online learning environments, learners engage in collaborative inquiry and meaning making through multiple forms of interaction [7]. These platforms enable self-determination in goal-setting and self-direction in managing learning processes [20], and they can also strengthen behavioural and emotional engagement [21]. Against this background, the IAM is widely regarded as a seminal framework for studying knowledge construction in computer-mediated settings [4]. This model provides a comprehensive framework of five stages that facilitate an understanding of how learners progressively construct knowledge through interaction. The five stages are as follows: 1) sharing and comparing information, 2) discovering and exploring dissonance or inconsistency among ideas, concepts or statements, 3) negotiating meaning/co-constructing knowledge, 4) testing and modifying a proposed synthesis or co-construction, and 5) agreeing with statement(s)/applying newly constructed meaning.

Notably, the original IAM was designed to examine knowledge construction that unfolds across multiple exchanges in structured discussions [7]. Subsequent research has extended this view and shows that the cognitive phases defined by the IAM can also be identified within a single message through linguistic and semantic features [1]. This is because, in online and social media contexts, comments often connect to a wider community conversation and shared cognitive context even when they are not part of an explicit reply chain [22], [23]. In such settings, each comment can be treated as a discursive contribution to collective knowledge construction, positioned within a dynamic and cumulative space of interaction rather than a single linear thread [1].

For example, using the IAM with discourse analysis, [2] showed that the informal learning context of YouTube science videos encouraged a shift from simple information exchange to argumentative negotiation, indicating higher-level knowledge construction. [3] adopted a revised IAM ato label phases of knowledge construction in TikTok comments. They found that interactions were predominantly social, while knowledge-related behaviours tended toward opinion sharing and the exploration of disagreements.

While the IAM has been applied across various platforms, analytical practices remain inconsistent. Differences in coding schemes, analytical units, and validity criteria have led to limited comparability across studies [24], [25]. Therefore, there is a need for more systematic and automated approaches to identify phases of knowledge construction with greater reliability and scalability across online learning contexts.

1.2 Automated Classification of Knowledge Construction↩︎

With the rapid advancement of natural language processing, deep learning models have been increasingly applied to the analysis and classification of knowledge construction in online learning environments [1]. Early studies have shown that neural architectures can effectively distinguish discourse segments corresponding to different phases of cognitive presence, demonstrating the feasibility of automated approaches for this task [12]. Transformer-based models such as BERT [14] and its successor DeBERTa, which introduces disentangled attention and enhanced contextual representations [15], have achieved substantial improvements in fine-grained discourse classification.

Applying BERT to the classification of cognitive presence in inquiry-based learning environments has yielded substantial agreement with human annotations, confirming the expressive advantage of Transformer models in fine-grained phase recognition [26]. A multi-label extension of BERT has further demonstrated robustness to interpretive ambiguity by capturing the fluid and overlapping nature of cognitive phases, tending to assign adjacent rather than distant categories [27]. Other studies have integrated semantic and behavioural indicators, such as attachments, tags, and glossary use, showing that textual information remains the dominant predictor, while additional behavioural traces offer only limited benefits [10]. Recent reviews have further highlighted both the potential of Transformer and large language models (LLMs) for educational discourse analysis and the need to ensure interpretability and cross-context robustness [28].

Building on the IAM, a deep learning framework grounded in its five-stage model of social knowledge construction achieved accuracies of 0.216 with Doc2Vec, 0.43 with a fine-tuned LLM, and 0.528 with a prompt-based LLM [1]. Their dataset comprised only 307 discussion posts, limiting generalisability. [3] also applied the IAM to classify knowledge construction phases in TikTok comments. Their dataset included 12,009 English-translated comments, but the distribution of labelled instances across the IAM phases was highly imbalanced. BERT achieved the highest macro and per-class F1 scores (0.66–0.77), outperforming logistic regression, multinomial Naïve Bayes, and SVM classifiers. Together, these studies highlight both the promise and the limitations of current Transformer-based approaches, which are constrained by single-language, small-scale, and imbalanced datasets.

Therefore, there is a need for studies that employ larger and more balanced multilingual annotated datasets, and that use transformer-based models with advanced training strategies to improve the robustness and accuracy of knowledge construction classification in online discussions.

1.3 Study Aim↩︎

This paper aims to propose a DeBERTa-v3-large–based classifier that integrates Focal Loss, label smoothing, and R-Drop regularization to improve the accuracy and robustness of knowledge construction phase classification in online discussions. Our contributions are threefold: (i) we introduce a transformer-based approach tailored for identifying phases of knowledge construction in online discussions; (ii) we benchmark our model against classical and transformer-based baselines on manually annotated discussion data; and (iii) we provide detailed ablation studies to assess the role of each training technique.

2 Model Structure↩︎

2.1 Overview↩︎

The proposed DeBERTa-KC architecture builds upon the DeBERTa-v3-large backbone to classify discourse segments into four phases of Knowledge Construction (KC): Non-KC, Share, Explore, and Negotiate. The framework integrates a transformer encoder with a composite optimization objective designed to enhance robustness under limited and imbalanced data conditions. Figure 1 illustrates the overall architecture.

2.2 Backbone Encoder↩︎

We employ the microsoft/deberta-v3-large model, a transformer architecture consisting of 24 layers, 16 self-attention heads per layer, and a hidden size of 1024. DeBERTa disentangles content and positional embeddings and applies a relative position bias during self-attention, which improves contextual representation over standard BERT and RoBERTa models [15]. The model processes each input comment as a sequence of tokens \(\{w_1, w_2, \dots, w_n\}\) tokenized using the SentencePiece tokenizer. Tokens are truncated to 256 tokens, padded to the maximum sequence length, and converted to embeddings before entering the transformer encoder.

The [CLS] token embedding \(h_{\text{CLS}}\) output by the final layer represents the entire comment and is passed to a classification head consisting of: \[\mathbf{z} = \text{Dropout}(h_{\text{CLS}}), \qquad \mathbf{p} = \text{Softmax}(\mathbf{Wz} + \mathbf{b})\] where \(\mathbf{W} \in \mathbb{R}^{4\times1024}\) and \(\mathbf{b}\in\mathbb{R}^4\) are learnable parameters. The probability vector \(\mathbf{p}\) represents the model’s predicted distribution over the four KC categories.

2.3 Training Objective↩︎

The loss function integrates three complementary components: Focal Loss, Label Smoothing, and R-Drop regularization. These elements jointly address class imbalance, overconfidence, and stochastic inconsistency during training.

2.3.0.1 Focal Loss with Label Smoothing.

We extend the standard cross-entropy loss with Focal Loss weighting [17] and label smoothing [18]. For a training instance with true label \(y\) and predicted probabilities \(\mathbf{p}\), the loss is defined as: \[\mathcal{L}_{FL} = - (1 - p_y)^\gamma \log(p_y')\] where \(\gamma=2.0\) controls the focusing strength, and \(p_y' = (1 - \epsilon)p_y + \frac{\epsilon}{K}\) applies label smoothing with \(\epsilon=0.05\) across \(K=4\) classes. This formulation down-weights easy examples and softens the target distribution to mitigate overfitting.

2.3.0.2 R-Drop Regularization.

R-Drop encourages prediction consistency between two stochastic forward passes of the same input by minimizing bidirectional Kullback–Leibler (KL) divergence [16]. Given two logits distributions \(\mathbf{p}_1\) and \(\mathbf{p}_2\), the R-Drop loss is: \[\mathcal{L}_{RD} = \frac{1}{2}\left[\text{KL}(\mathbf{p}_1 \| \mathbf{p}_2) + \text{KL}(\mathbf{p}_2 \| \mathbf{p}_1)\right]\] which penalizes divergence due to dropout or attention randomness.

2.3.0.3 Overall Objective.

The total loss is a weighted sum: \[\mathcal{L} = \mathcal{L}_{FL} + \lambda_{RD} \mathcal{L}_{RD}\] where \(\lambda_{RD}=1.0\) empirically balances the main and consistency terms. During training, gradients are backpropagated jointly through the encoder and classification layers.

2.4 Optimization and Regularization↩︎

Training employs the AdamW optimizer with a cosine learning rate scheduler, initial learning rate \(1\times10^{-5}\), warm-up ratio of 0.1, and weight decay of 0.05. Each fold runs for up to 10 epochs with early stopping (patience = 2). Batch sizes are 8 for training and 16 for evaluation, with mixed precision (FP16 or BF16) for computational efficiency. Cross-validation is stratified (10-fold) to ensure balanced representation of KC categories, and all folds are ensembled at inference by averaging softmax probabilities.

2.5 Inference and Ensemble↩︎

At inference, predictions from each fold are averaged to form an ensemble probability distribution: \[\hat{\mathbf{p}} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{p}^{(i)}\] where \(N=4\) is the number of folds. The class with the highest averaged probability is selected as the final label. This ensemble strategy improves robustness and stabilizes per-class metrics across folds.

3 Methods↩︎

3.1 Overview↩︎

Figure 2 illustrates the end-to-end pipeline used in this study, covering data collection, preprocessing, model training, evaluation, and reproducibility procedures. The design follows an orchestrated workflow encompassing five phases—Collect, Capture, Ingest, Compute, and Store & Use—each representing a distinct step in the experimental lifecycle. The pipeline was implemented using open-source tools to ensure transparency and reproducibility, with orchestration handled through modular Python scripts and all model training, evaluation, and logging automated using GPU-enabled experiment tracking.

3.2 Knowledge Construction Coding↩︎

To capture the specific characteristics of learner discourse in online informal learning environments, we adopted the Knowledge Construction (KC) framework proposed by Nguyen and Diederich [3], which extends the original Interaction Analysis Model (IAM). The revised framework introduces a non-knowledge construction (nonKC) category to account for comments with primarily social or affective intentions. Previous studies [2], [7] have shown that the upper-level IAM categories—Test/Modify Proposed Ideas and Apply New Knowledge—rarely appear in user-generated content and often suffer from extreme data imbalance. Therefore, following Nguyen and Diederich [3], these two categories were merged into the broader Negotiate category.

The final taxonomy thus comprised four KC categories: nonKC, Share, Explore, and Negotiate, representing an increasing level of epistemic engagement. Each category was defined operationally with examples to guide annotation (Table 1). This coding scheme captures both cognitive and dialogic aspects of online discourse while maintaining practical balance for supervised learning.

3.3 Dataset Construction↩︎

3.3.0.1 Data Collection.

We collected viewer comments from four high-engagement science communication YouTube channels—MuseumScience, SciShow, SickScience, and WorldScience—using the YouTube Data API v3 between 2022 and 2024. These channels were selected based on three criteria: (1) high posting frequency, (2) large subscriber base, and (3) diversity of scientific topics. A total of 2,198 videos were retrieved, including 1,223 short-form videos and 975 long-form videos. From these, 609,763 top-level comments were extracted, comprising 299,637 from short videos and 310,126 from standard videos.

3.3.0.2 Annotation.

The annotation process followed a stratified and iterative sampling strategy to ensure category balance and reduce coder drift. In nine independent sampling rounds, Coder A manually labeled 5,000 unique comments per round using the KC framework, oversampling underrepresented categories. Coder B subsequently reviewed and refined all labels, and only comments with inter-coder agreement were retained. The final balanced dataset contained 5,000 samples per category (20,000 total), with equal representation from short and long videos. Inter-coder reliability reached a Cohen’s \(\kappa\) of 0.87, indicating strong agreement.

3.3.0.3 Preprocessing.

Comments were tokenized, lowercased, and normalized by removing hyperlinks, emojis, and user mentions while preserving punctuation relevant to discourse structure. Categories were then normalized across the combined corpus, and stratified 10-fold cross-validation splits were generated using scikit-learn’s StratifiedKFold to maintain label distribution across folds.

3.4 Model Baselines↩︎

To benchmark performance, we compared classical feature-based classifiers with transformer-based encoders. All experiments were conducted under identical cross-validation folds.

3.4.0.1 Classical Baselines.

Two baseline models were implemented using scikit-learn:

1. TF–IDF + Logistic Regression, using character-level \(n\)-grams (range 3–5) with balanced class weighting.

2. TF–IDF + Linear SVM, using the same representation with a linear kernel (LinearSVC).

3.4.0.2 Transformer Baselines.

We fine-tuned DistilBERT (base), RoBERTa-base, and DeBERTa-v3-base models using the Hugging Face Transformers library. Each model was trained under identical optimization settings to ensure fair comparison. These baselines served as reference points for evaluating our proposed DeBERTa-KC architecture.

3.5 Proposed Model: DeBERTa-KC↩︎

The proposed DeBERTa-KC extends DeBERTa-v3-large by incorporating three regularization strategies: (1) Focal Loss to address class imbalance by emphasizing harder-to-classify samples, (2) Label Smoothing (0.1) to improve calibration and reduce overconfidence, and (3) R-Drop to minimize prediction variance across stochastic forward passes. This configuration enhances robustness and generalization across folds. The model was trained and evaluated on NVIDIA RTX A6000 GPUs, using PyTorch mixed-precision (FP16) training.

3.6 Training and Optimization↩︎

Transformer models were fine-tuned using the Trainer API with the following hyperparameters: maximum sequence length of 256, AdamW optimizer with cosine learning rate decay, initial learning rate of \(1\times10^{-5}\), weight decay of 0.05, warm-up ratio of 0.1, and early stopping with a patience of 2 epochs. The training batch size was 8 and the evaluation batch size 16. For reproducibility, all random seeds were fixed across runs and training logs were captured through automated experiment tracking.

3.7 Evaluation Protocol↩︎

Model performance was assessed using 10-fold stratified cross-validation. For each fold, models were trained on nine folds and evaluated on the remaining one. Reported metrics include accuracy, macro-F1, and weighted-F1, alongside per-class precision, recall, and F1-score. Each metric is expressed as mean \(\pm\) standard deviation (SD) across folds.

3.8 Statistical and Reproducibility Analysis↩︎

To assess significance, paired \(t\)-tests and Wilcoxon signed-rank tests were applied to fold-level macro-F1 distributions between DeBERTa-KC and baseline models, using \(p < 0.05\) as the threshold. Effect sizes were computed via Cohen’s \(d\). Variance homogeneity was verified through Levene’s test, and confidence intervals (95%) were estimated using bootstrap resampling.

All experiments, configurations, and outputs (e.g., per-fold metrics, aggregated summaries, and model comparison dashboards) were automatically logged and versioned. This ensures reproducibility and enables transparent comparison between classical and transformer-based approaches. The results and model artifacts are stored in structured formats (.csv, .json) as shown in Figure 2.

4 Results and Analysis↩︎

This section reports the experimental results of the proposed DeBERTa-KC model and baseline systems for automatic classification of knowledge construction (KC) phases in online learning discourse. All results are averaged over ten stratified cross-validation folds, ensuring that each fold maintains the same label distribution. The analyses presented below address the overall model performance, per-category classification behaviour, ablation experiments, and the outcomes of significance and agreement testing.

4.1 Overall Model Comparison↩︎

Table 2 presents the overall accuracy and F1-scores for both classical and transformer-based models. The classical baselines using TF–IDF representations achieved reasonable performance (macro-F1 \(\approx 0.72\)), confirming that character-level \(n\)-grams can capture surface-level lexical regularities present in learning discourse. However, these representations lacked contextual awareness, leading to moderate performance variability across folds.

Transformer models, by contrast, demonstrated markedly superior results. DistilBERT (base) achieved a macro-F1 of \(0.807 \pm 0.010\), while RoBERTa-base improved further to \(0.821 \pm 0.009\). The stronger DeBERTa-v3-base model reached \(0.834 \pm 0.007\), validating the benefit of disentangled attention mechanisms and enhanced masked decoding in capturing nuanced relationships among tokens. Our proposed DeBERTa-KC configuration (DeBERTa-v3-large + Focal Loss + Label Smoothing + R-Drop) achieved the best overall result with \(0.836 \pm 0.008\) macro-F1, outperforming all smaller backbones. When Label Smoothing was removed, the mean macro-F1 slightly increased to \(0.841 \pm 0.009\), though with a modest rise in variance, indicating that Label Smoothing enhances cross-fold consistency at a marginal cost to the peak performance.

Figure 3 visualizes the distribution of fold-level macro-F1 values. The figure demonstrates a clear upward trend in mean performance from classical to transformer-based approaches, accompanied by a reduction in variance. Among all configurations, DeBERTa-KC exhibits both the highest mean performance and the lowest dispersion, underscoring its stable generalization across folds.

4.2 Per-Category Performance↩︎

Tables 3–6 provide detailed per-category precision, recall, and F1-scores. Across all four knowledge construction labels—Negotiate, Share, Explore, and NonKC—transformer-based models consistently outperformed the classical baselines.

For the Negotiate category, all transformer architectures achieved strong and stable performance, with F1-scores close to 0.90. This indicates a robust capacity to detect discourse segments in which participants collaboratively refine and integrate ideas. DeBERTa-KC attained an F1 of \(0.897 \pm 0.012\), closely matching the highest-scoring no-Label-Smoothing variant (\(0.900 \pm 0.006\)). Such consistency reflects the model’s ability to capture deep semantic cues that distinguish genuine negotiation from more superficial forms of interaction.

The Share category exhibited a substantial gain in performance from classical to transformer models, increasing from \(0.648 \pm 0.017\) (TF–IDF + Logistic Regression) to \(0.793 \pm 0.009\) with DeBERTa-KC. This improvement highlights the model’s enhanced ability to recognise comments that focus on exchanging information or personal experiences relevant to the learning context. The no-Label-Smoothing variant reached the highest performance (\(0.797 \pm 0.016\)), suggesting that sharper decision boundaries sometimes enhance the recognition of sharing-oriented statements.

The Explore category, which encompasses discourse reflecting analytical reasoning and hypothesis generation, benefited the most from transformer-based contextual representations. The F1-score rose from \(0.686 \pm 0.023\) with TF–IDF + Logistic Regression to \(0.817 \pm 0.015\) with DeBERTa-KC. The variant without Label Smoothing achieved the highest recall (\(0.850 \pm 0.030\)), implying that clearer boundary learning may aid in identifying exploratory discourse patterns, which often exhibit subtle lexical and syntactic variation.

Finally, the NonKC category—representing social, affective, or organisational comments that do not contribute directly to knowledge construction—was also handled effectively by transformer-based architectures. DeBERTa-KC obtained \(0.838 \pm 0.012\) F1, outperforming all baselines. Its no-Label-Smoothing variant achieved the highest recall (\(0.826 \pm 0.035\)), confirming that the model accurately distinguishes socially oriented statements from knowledge-building exchanges.

Overall, DeBERTa-KC improved both precision and recall across all four categories, particularly for the semantically complex Explore and Negotiate labels. The model’s architecture and loss design thus appear to enhance contextual discrimination and mitigate bias toward dominant classes.

4.3 Ablation Study↩︎

An ablation analysis was conducted to examine the individual contribution of each component in the DeBERTa-KC configuration. Removing any of the auxiliary modules resulted in either a decrease in macro-F1 or an increase in cross-fold variance. Focal Loss improved the recall of minority categories by dynamically scaling the gradient contribution of hard-to-classify samples, thereby mitigating class imbalance. Label Smoothing enhanced training stability by preventing the model from becoming overconfident in its predictions, which is particularly valuable in small, imbalanced datasets. The R-Drop regularisation further reduced overfitting by enforcing consistency between stochastic forward passes. Together, these mechanisms produced a model that balances accuracy, robustness, and interpretability.

4.4 Statistical Significance and Effect Size Analysis↩︎

To assess the statistical significance of performance differences across models, non-parametric Friedman and pairwise Wilcoxon signed-rank tests were performed on the fold-level macro-F1 distributions (Table 7). The Friedman test indicated a significant overall difference among models (\(\chi^2 = 39.82\), \(p < 0.001\)). Subsequent pairwise Wilcoxon tests, with Holm correction for multiple comparisons, confirmed that all transformer models significantly outperformed the TF–IDF baselines (\(p < 0.01\)). DeBERTa-KC also showed significant improvements over DistilBERT and RoBERTa-base (\(p = 0.002\) in both cases). The performance difference between DeBERTa-KC and its base backbone, DeBERTa-v3, was modest but statistically detectable (\(p = 0.0074\), Cohen’s \(d = 0.89\)), indicating a small yet consistent performance gain.

Cohen’s \(d\) effect size analysis further revealed large effects (\(d > 0.8\)) between the classical and transformer models, moderate effects (\(0.5 < d < 0.8\)) between successive transformer generations, and small effects (\(|d| < 0.2\)) among DeBERTa-KC ablations. These findings suggest that the proposed configuration contributes measurable yet controlled improvements over already strong baselines. Levene’s variance test (\(W = 3.02\), \(p = 0.027\)) indicated slightly heterogeneous variances, primarily due to the higher variability of the classical models, whereas transformer-based configurations exhibited stable dispersion. Bootstrapped confidence intervals confirmed the narrowest 95% bounds for DeBERTa-KC ([0.836, 0.839]), demonstrating strong generalization stability compared with broader intervals observed for TF–IDF baselines ([0.725, 0.735]).

4.5 Agreement and Robustness Analysis↩︎

To further evaluate the consistency between the base and proposed transformer variants, a Bland–Altman agreement analysis was conducted using fold-level macro-F1 values from DeBERTa-v3 and DeBERTa-KC (Figure 4). Each point represents a cross-validation fold, plotted as the mean of both models’ scores on the x-axis and their difference on the y-axis. The analysis revealed a mean bias of \(-0.0031\), indicating that DeBERTa-KC achieved slightly higher performance on average. The 95% limits of agreement (±0.006) demonstrate that nearly all fold-level differences lie within a narrow and symmetric range around zero, confirming high agreement and consistent improvement across folds. No systematic bias was observed across performance ranges, suggesting that DeBERTa-KC’s enhancements are general rather than fold-specific. Combined with the distributional patterns in Figure 3, these results confirm that the proposed model exhibits both higher mean performance and greater reliability than its base counterpart.

4.6 Summary of Findings↩︎

In summary, the DeBERTa-KC model achieved superior overall performance relative to all baselines, with the highest mean macro-F1 and the lowest cross-validation variance. It demonstrated balanced classification across all four KC categories, particularly improving the recognition of higher-order discourse such as negotiation and exploration. The integration of Focal Loss, Label Smoothing, and R-Drop enhanced both stability and robustness, while statistical testing confirmed the significance and consistency of these improvements. The agreement analysis further validated that DeBERTa-KC provides consistent and reproducible gains over the base transformer, thereby establishing a reliable framework for automated knowledge construction analysis in large-scale online learning environments.

5 Discussion↩︎

The findings demonstrate that the proposed DeBERTa-KC model substantially advances the automated identification of knowledge construction (KC) levels in online learning discourse. The consistent performance improvements across folds and categories indicate that the architecture effectively captures both the lexical and contextual features underlying epistemic engagement in user-generated comments. This section discusses the results in relation to prior studies, the pedagogical implications of the approach, and its methodological and computational contributions.

5.1 Performance and Model Behaviour↩︎

The results clearly indicate that transformer-based architectures outperform traditional feature-based models by a considerable margin. Classical baselines such as TF–IDF with Logistic Regression or Linear SVM achieve moderate performance (macro-F1 \(\approx\) 0.72), confirming that surface-level lexical patterns provide a useful but limited signal of epistemic activity. In contrast, transformer encoders capture deeper discourse semantics, yielding macro-F1 scores above 0.80 across all transformer variants. Among them, DeBERTa-KC achieves the highest and most stable performance (\(0.836 \pm 0.008\) macro-F1), surpassing both DistilBERT and RoBERTa-base by statistically significant margins. These gains highlight the effectiveness of disentangled attention mechanisms and the regularization strategies (Focal Loss, Label Smoothing, and R-Drop) in enhancing generalization.

The per-category analysis further supports this observation. The model performs most strongly on the Negotiate and Explore categories, where discourse involves elaboration, reasoning, and conceptual integration—features that benefit from deep contextual embeddings. These categories are cognitively demanding, often involving subtle rhetorical markers and reasoning structures, and thus challenge simpler models that rely primarily on local lexical cues. The DeBERTa-KC model’s capacity to model long-range dependencies and contextual coherence enables it to discern these dialogic nuances more reliably. Although the Share and nonKC categories exhibit slightly lower recall, this aligns with their lexical overlap and shorter sentence structure, which reduce discriminative information. Nevertheless, the model maintains robust F1-scores across all categories, suggesting balanced classification performance.

5.2 Interpretation of Statistical Analyses↩︎

The statistical and robustness analyses provide further insight into the reliability of these results. The Friedman and Wilcoxon tests confirmed that the observed improvements of DeBERTa-KC over baseline models are statistically significant (\(p < 0.01\)). The computed Cohen’s \(d\) values indicate large effect sizes when comparing classical to transformer-based approaches (\(d > 2.7\)) and medium effects among transformer generations (\(1.0 < d < 1.6\)). The lack of significant difference between DeBERTa-KC and its ablations (\(p > 0.3\)) demonstrates that each regularization component contributes incrementally rather than producing overfitted gains.

Moreover, the Bland–Altman analysis between DeBERTa-v3 and DeBERTa-KC shows a minimal mean bias (\(-0.0031\)) and narrow limits of agreement, indicating that performance improvements are both consistent and replicable across folds. This pattern reinforces the model’s stability rather than stochastic variation. The 95% confidence interval of the macro-F1 ([0.829, 0.843]) further supports the robustness of DeBERTa-KC’s predictions. Together, these findings demonstrate that the proposed enhancements yield genuine and reproducible gains without compromising model variance.

5.3 Implications for Knowledge Construction Analysis↩︎

From an educational perspective, these findings have important implications for understanding and supporting knowledge construction in informal digital learning environments. Unlike structured online courses, YouTube discussions are open, decentralized, and conversationally dynamic. Automatically identifying KC levels within such contexts enables large-scale analysis of how learners engage epistemically with scientific content. The ability to reliably distinguish between nonKC (social talk) and higher-level categories such as Negotiate provides a foundation for evaluating the depth and quality of learner discourse at scale.

Moreover, the model’s interpretability through attention visualization (not shown here) suggests that contextual phrases and argumentative connectors (e.g., “because”, “I think”, “for example”) play a central role in predicting higher KC levels. This aligns with prior qualitative findings [2], [7] that link epistemic markers to conceptual elaboration and co-construction of meaning. Therefore, integrating models such as DeBERTa-KC into learning analytics systems could offer educators empirical indicators of cognitive engagement, helping them scaffold discourse quality and guide moderation strategies in open learning spaces.

5.4 Methodological and Computational Contributions↩︎

Beyond performance metrics, this study contributes a reproducible computational framework for discourse analysis in open social learning platforms. The orchestrated workflow (Figure 2) ensures end-to-end transparency, covering data acquisition, annotation, preprocessing, training, and statistical validation. By combining GPU-based transformer fine-tuning with automated experiment tracking, the approach aligns with emerging standards in educational data science for reproducibility and open experimentation. All results, including cross-validation metrics, model checkpoints, and statistical outputs, are stored in structured formats to facilitate replication and extension.

Furthermore, the use of balanced sampling and cross-validation provides a more reliable estimate of generalization than previous single-split studies. The inclusion of Focal Loss and R-Drop in educational text classification introduces methodological innovation by addressing class imbalance and overfitting—issues commonly overlooked in learning analytics applications. The ablation experiments confirm that these mechanisms collectively stabilize the learning process without inflating scores artificially.

5.5 Limitations and Future Work↩︎

Despite its contributions, several limitations should be acknowledged. First, the dataset focuses on English-language science discourse, which limits generalizability across linguistic and cultural contexts. Future research should evaluate cross-lingual transferability using multilingual transformer backbones (e.g., XLM-R, mDeBERTa). Second, while the annotation process achieved high inter-coder reliability, manual labeling remains resource-intensive. Semi-supervised or active learning strategies could be explored to expand labeled data more efficiently. Third, although the model identifies KC levels effectively, it does not explicitly capture the interactional dynamics or temporal progression of discourse. Integrating sequential or graph-based modeling could help trace how knowledge construction evolves across comment threads.

Finally, interpretability remains a key challenge for educational deployment. Future work should investigate how attention or gradient-based attribution can provide transparent explanations suitable for educators, aligning automated predictions with theoretical constructs from educational discourse analysis.

6 Conclusion↩︎

This study proposed and evaluated DeBERTa-KC, a transformer-based model designed to automatically classify knowledge construction (KC) levels in large-scale online science learning discourse. By integrating Focal Loss, Label Smoothing, and R-Drop regularization into the DeBERTa-v3 architecture, the model achieved substantial and statistically significant improvements over classical and baseline transformer approaches. Across 10-fold cross-validation, DeBERTa-KC attained stable performance (macro-F1 \(= 0.836 \pm 0.008\)), demonstrating robust generalization across diverse types of learner comments and discourse categories.

Beyond quantitative performance, the study contributes a reproducible computational framework that bridges natural language processing with educational discourse analysis. The orchestrated workflow—from YouTube comment collection and manual annotation to model training, evaluation, and reproducibility tracking—provides an end-to-end methodological template for future research. The model’s ability to distinguish between social (nonKC) and epistemically engaged (Share, Explore, and Negotiate) comments highlights its capacity to capture fine-grained indicators of cognitive engagement in informal learning environments.

From an educational perspective, this work demonstrates the feasibility of using large language models to examine how learners construct, share, and negotiate scientific ideas in open digital ecosystems. Automating KC classification enables scalable monitoring of discourse quality, which may inform learning analytics dashboards, teacher interventions, or automated feedback systems in future implementations. The insights derived from such models can support evidence-based strategies for promoting epistemic dialogue and metacognitive reflection in informal and hybrid learning settings.

Several avenues remain for further exploration. Future research should extend this framework to multilingual and multimodal contexts, incorporating cross-lingual representations or audio–text alignment from video transcripts. Integrating sequential or interaction-aware modeling could also uncover temporal dynamics of knowledge construction across threads. Finally, advancing model interpretability through attention visualization or explainable AI techniques will be essential for ensuring transparency and pedagogical trustworthiness when applying such systems in educational practice.

In conclusion, the proposed DeBERTa-KC model offers a theoretically grounded, empirically validated, and computationally reproducible approach for analyzing knowledge construction in large-scale online learning discourse. By combining deep language modeling with educational theory, this work provides a pathway toward scalable, interpretable, and ethically informed AI applications in the study of learning and knowledge-building processes.

References↩︎

---

1. Citation: Authors. Title. Pages.… DOI:000000/11111.↩︎