1 Introduction↩︎

Mitigating weaknesses has become increasingly necessary as software systems grow in complexity and adversaries adopt sophisticated attack techniques. Software and hardware weaknesses cataloged in the CWE[1] enable systematic vulnerability assessment, mitigation planning, and security decision making[2]. The CWE taxonomy organizes weaknesses into abstraction levels such as Base, Class, and Pillar, where upper-level nodes capture broad concepts, and lower-level nodes represent specific details. In practice, a small number of weakness types dominate the corpus, while many rare but high-impact categories remain significantly underrepresented.

Transformer-based encoders such as BERT [3] and RoBERTa [4] have become strong baselines for CWE text classification, capturing bidirectional contextual dependencies that outperform earlier Convolutional Neural Network (CNN) [5] and Recurrent Neural Network (RNN) architectures [6]. However, class imbalance remains a fundamental challenge that degrades performance on rare CWE categories, and CWE labels carry explicit parent-child dependencies [7]. For instance, a Variant is always a child of a Base, meaning misclassification at higher abstraction levels cascades downward to finer-grained classes.

Research on hierarchical classification emphasizes that maintaining parent-child relationships is crucial for building reliable prediction models [7]. Traditional approaches, such as random oversampling and undersampling [8], rebalance label distributions but risk overfitting on minority samples or losing valuable information from the majority class. More advanced methods, SMOTE [9] and ADASYN [10], generate synthetic minority samples in feature space. SMOTE interpolates between minority instances to reduce classifier bias toward majority labels, while ADASYN concentrates synthesis on harder boundary regions where misclassifications are likely. The specific contributions of our work are as follows:

1. We empirically evaluate ML (RF, SVM), deep learning (CNN, BiGRU), and transformer-based (BERT) models under SMOTE and ADASYN resampling on an imbalanced hierarchical CWE dataset.

2. We demonstrate that synthetic interpolation in high-dimensional embedding spaces violates CWE parent-child constraints, marginally helping classical ML models while consistently degrading deep learning models.

3. We introduce a Hierarchy-Aware RoBERTa model that injects CWE structural priors through learnable parent-class embeddings for end-to-end joint modeling of semantic content and hierarchical structure.

4. Our proposed model achieves a weighted F1-score of \(0.76\) without augmentation, outperforming all baselines with Class F1 improving from \(0.49\) to \(0.60\) over BERT.

This paper is structured into the following sections. Section 2 presents the related work. Section 3 presents the technical background of the models. Sections 4 and 5 contain the methodology and experimental setup, respectively. Section 6 presents the results. Section 7 presents the discussion and limitations. Section 8 concludes the paper with future work.

2 Related Work↩︎

2.1 CWE-Based Vulnerability Classification↩︎

Prior work primarily treats CWE classification as a supervised multiclass problem in data-rich settings. Chhetri et al. [2] showed that BERT outperformed CNN, LSTM, and HAN architectures in predicting the consequences of cyber attacks from the descriptions of CWE, but performance degraded under sparse and imbalanced conditions. VulnBERTa [11], a hierarchical RoBERTa-based classifier trained on reports from the National Vulnerability Database (NVD), achieved strong accuracy on frequent CWE classes but significant degradation on rare labels. Contreras et al. [12] evaluated BI-LSTM and BiGRU models on the Software Assurance Reference Dataset (SARD) and the NVD with strong overall results but struggled to distinguish semantically overlapping CWE classes, highlighting the persistent barrier of long-tail label distributions.

2.2 Semantic and Hierarchical Classification↩︎

Previous work [13] applied BERT-based cross-encoders with binary chaining to the CWE View-1003 hierarchy, improving accuracy on high-cardinality label sets but remaining limited in tail classes. V2W-BERT [14] demonstrated that label embeddings and textual CWE definitions improve representations for single-label CWE assignment. Classical ML approaches including Random Forests (RF) [15] and Support Vector Machines (SVM) [16] perform well on frequent weaknesses but consistently degrade on rare ones, motivating approaches that explicitly model label structure.

2.3 Class Imbalance in Datasets↩︎

SMOTE [9] and ADASYN [10] have proven effective for classifiers based on classical ML and CNN, and recent work has incorporated oversampling directly into deep learning training loops [17]. However, the behavior of SMOTE and ADASYN on hierarchically structured CWE text embeddings, where parent-child constraints impose structural dependencies that linear interpolation cannot respect, remains largely unexplored, a gap this paper directly addresses.

3 Technical Background↩︎

3.1 Machine Learning Models↩︎

3.1.1 Random Forest↩︎

(RF)[15] constructs an ensemble of decision trees on bootstrapped subsets of training data, aggregating predictions to reduce variance and overfitting. Each tree recursively selects splits that maximize impurity reduction, performing well on structured feature spaces, but struggling on high-dimensional sparse text representations. Here, the best split is chosen as the feature and threshold that leads to the largest decrease in impurity between the parent and its children.

3.1.2 Support Vector Machine↩︎

(SVM) [16] learns a maximum-margin hyperplane that separates classes in a high-dimensional feature space. A linear kernel is well suited for sparse text representations, making SVM a strong baseline for short-text classification tasks such as CWE descriptions.

3.2 Deep Learning Models↩︎

3.2.1 Convolutional Neural Networks↩︎

(CNN) [5] applies 1-D convolutional filters over token embeddings to capture local n-gram patterns, followed by max-pooling to retain the most salient activations. While effective for short range dependencies, CNNs are sensitive to perturbations in embedding space introduced by synthetic oversampling.

3.2.2 Bidirectional Gated Recurrent Unit↩︎

(BiGRU)[6] processes sequences in both forward and backward directions, concatenating hidden states at each position to capture full contextual dependencies. The gated mechanism controls information flow across time steps, making BiGRU effective for modeling order-sensitive vulnerability descriptions.

3.3 Transformers Based Models↩︎

3.3.1 Bidirectional Encoder Representations from Transformers↩︎

(BERT) [3] uses stacked encoder layers with multi-head self-attention, pretrained via masked language modeling. Robustly Optimized Bidirectional Encoder Representations from Transformers (RoBERTa) [4] improves on BERT by removing next sentence prediction, adopting dynamic masking, and training on substantially larger corpora, yielding stronger contextual representations across NLP benchmarks. SecureBERT [18] extends RoBERTa through continuous pretraining on a large cybersecurity corpus and serves as the encoder backbone for our proposed model.

4 Methodology↩︎

4.1 Oversampling↩︎

The dataset exhibited severe class imbalance across five classes, i.e., Base (393), Variant (219), Class (83), Compound (8), and Pillar (5). Oversampling was applied exclusively to the training set to prevent data leakage. SMOTE equalized all classes to 393 samples using \(k\)-neighbors\(=2\), while ADASYN produced near parity counts (Compound 395, Base 393, Pillar 392, Class 387, Variant 362) by concentrating synthesis on harder boundary regions. Figure 1 illustrates the resulting class distributions before and after resampling.

4.2 Model Architectures↩︎

For classical baselines, RF was set with 100 decision trees (\(n\)-estimators \(= 100\)) and a fixed random seed (random-state \(= 42\)). The SVM classifier was configured to use a linear kernel, which is appropriate in high-dimensional, sparse feature representations.

For deep learning, CNN used 128 convolutional filters of kernel sizes [4,6,6] with ReLU activation and adaptive max pooling, while BiGRU used a hidden dimension of 128 with stacked recurrent layers to capture long-range contextual dependencies. Both models mapped tokens to 256-dimensional embeddings trained end-to-end, fusing a 16-dimensional categorical status embedding via a 128-dimensional fusion layer (ReLU, dropout 0.2). BERT-base (uncased) used the [CLS] pooled output fused with a 16-dimensional status embedding, followed by a 128-dimensional linear layer (ReLU, dropout 0.2). All models were trained with a batch size of 8 for 15 epochs.

4.3 Hierarchy-Aware RoBERTa↩︎

We propose a Hierarchy-Aware RoBERTa architecture (Algorithm 3) that explicitly incorporates label hierarchy information into Transformer-based text classification. A pretrained SecureBERT encoder[18] produces a 768-dimensional [CLS] representation from the input text. To inject hierarchical knowledge, we introduce a learnable parent-category embedding that captures coarse-grained structural relationships among labels. Both parent categories are projected to a 24-dimensional dense embedding, trained together with the rest of the network.

The BERT pooled representation is then fused with the parent embedding, resulting in a unified feature representation. This represents both semantic content and hierarchical context. This fused representation is regularized using dropout with a rate of 0.3 before being passed to a linear classification head that outputs logits over the target classes.

Figure 2 illustrates the proposed Hierarchy-Aware RoBERTa architecture, which combines semantic representations from a pretrained RoBERTa encoder [18] with learned parent-label embeddings. The textual embedding and hierarchical context are fused via concatenation and passed through a dropout-regularized linear classifier to produce fine-grained child class predictions. This design enables hierarchy-informed classification by jointly leveraging contextual language understanding and structured label information. To ensure consistency across all models, deep learning models (CNN, BiGRU, BERT), and our proposed Hierarchy-Aware RoBERTa were all trained using the same batch size of 8 and 15 epochs.

5 Experimental Procedure↩︎

5.1 Dataset↩︎

The dataset used in this study was sourced from an enhanced version of the MITRE Common Weakness Enumeration (CWE) dataset. The CWE repository is continuously updated and maintained to incorporate newly identified software vulnerabilities and attack patterns. The dataset¹ contains 944 data points with 23 columns. For the experiments, we selected three columns (i.e., CWE-ID, Name, Weakness Abstraction, Description). The Weakness Abstraction has five unique values as presented in Figure 4. The dataset includes five classes in “\(Weakness Abstraction\)” column, Base sample of 524, Variant sample of 292, Class sample of 111, Pillar sample of 10, and a Compound sample of 7.

5.2 Data Processing↩︎

In this experiment, all five abstraction classes were retained for classification. The Name and Description fields were concatenated using a [SEP] token after removing noise patterns and trimming whitespace. Figure 5 illustrates a sample of the processed dataset. Labels from Weakness Abstraction were encoded as integers (0–4) and tokenized using BertTokenizer. Data were split 75:25 for training and testing, with all inputs padded or truncated to a maximum sequence length of 128 tokens.

6 Results↩︎

6.1 Model Accuracy↩︎

Figure 6 compares the performance of various machine learning, deep learning, and transformer-based models in terms of their accuracy as the main metric. RF achieved a baseline accuracy of 0.65, improving its performance to 0.69 with both SMOTE and ADASYN. SVM performed better than RF with an accuracy of 0.72, while SMOTE and ADASYN demonstrated the same performance, i.e., 0.71 and 0.72, respectively. Such findings suggest that oversampling is weakly helpful with classical models and helps address class imbalance. Similarly, the vanilla CNN dropped from 0.71 to 0.55 with SMOTE and 0.51 with ADASYN, while BiGRU fell from 0.70 to 0.53 and 0.44, respectively, confirming that oversampling can introduce noise that adversely impacts sequence-based and convolutional frameworks. BERT achieved an accuracy of 0.74, while the proposed Hierarchy-Aware RoBERTa achieved the highest accuracy of 0.76, demonstrating the advantage of structural priors over synthetic data generation.

6.2 Classification Performance↩︎

Table 1 reports the precision, recall, and F1-score per-class across all models and augmentation conditions. The improvement was primarily observed in minority classes. For example, the Class F1-score increased from 0.07 to 0.42 with SMOTE and 0.52 with ADASYN for RF. However, Compound and Pillar remained at 0.00 F1 across all settings. SVM showed stronger overall performance with a weighted F1 of 0.71, maintaining stable scores of 0.70-0.71 under both oversampling methods, indicating robustness on well-represented classes.

For deep learning models, CNN achieved a baseline weighted F1 of 0.68, with strong performance on Base (F1\(=0.79\)) and Variant (F1\(=0.67\)). However, applying SMOTE and ADASYN reduced the performance to 0.60 and 0.56, respectively, with Class F1 dropping to as low as 0.08 under ADASYN. BiGRU achieved a baseline weighted F1 of 0.69, dropping to 0.58 with SMOTE and 0.52 with ADASYN, with substantial degradation in minority classes, highlighting the sensitivity of sequence-based models to oversampling-induced noise.

Transformer-based BERT achieved a strong baseline weighted F1 of 0.73, driven by high performance on Base (F1\(=0.78\)) and Variant (F1\(=0.78\)). No augmentation was applied for BERT, as its contextual embeddings were sufficient without synthetic oversampling. Our Hierarchy-Aware RoBERTa further improved performance, achieving the highest weighted F1 of 0.76, consistently outperforming all models across Base (F1\(=0.81\)), Class (F1\(=0.60\)), and Variant (F1\(=0.76\)). In particular, Compound and Pillar remain at 0.00 F1 in all models, reflecting the persistent challenge of supervised learning under extreme data sparsity.

7 Discussion↩︎

Although data augmentation techniques such as SMOTE and ADASYN are widely used to address class imbalance in classical machine learning, they are not well suited for transformer-based models. BERT does not use fixed, low-dimensional feature vectors, but learns contextualized, high-dimensional semantic representations directly from raw text, meaning that synthetic interpolation produces samples that lack linguistic coherence and add noise rather than signal. For CNN and BiGRU models, small perturbations in embedding space correspond to large taxonomic shifts, causing models to overfit to synthetic artifacts and degrading generalization performance on minority and hierarchical classes.

Furthermore, SMOTE and ADASYN do not conserve the hierarchical relationship that exists within the data. Classes such as Variant are semantically dependent on their parent classes (e.g., Base). Synthetic oversampling does not respect such constraints and can produce samples that are inconsistent with their underlying class structure and confuse the model during training. In the CWE taxonomy, each entry links to a Related Weakness via an explicit parent-child chain, meaning errors at higher abstraction levels cascade downward to finer-grained classes, a structural dependency that oversampling methods ignore entirely by treating each sample as an independent entity in a flat feature space. CWEs are not isolated texts but are organized as a tree structured graph where more abstract weaknesses act as semantic parents to narrower ones, and interpolating between embeddings from different parent nodes produces representations of non-existent or indistinct weaknesses, giving statistically balanced but structurally invalid embeddings.

Our Hierarchy-Aware RoBERTa model explicitly incorporates structural knowledge by introducing a learnable parent embedding. In the CWE taxonomy, weakness abstractions are strictly hierarchical. For instance, a "Variant" class is semantically and structurally dependent on its corresponding "Base" parent. To model this dependency, we formulate the classification task not as an independent prediction \(P(Y_{child}|X_{text})\), but as a conditional probability problem \(P(Y_{child}|X_{text}, Y_{parent})\). We utilize "Parent ID" as a structural prior to guide the model to learn that if the parent is a Base type, the child is likely a Variant. We define two input vectors: Text Vector (\(V_{text}\)): The output from BERT (768 dimensions). Parent Vector (\(V_{graph}\)): A learnable embedding for the parent ID (24 dimensions). The model fuses them via concatenation:\(V_{final} = \text{Concat}(V_{text}, V_{graph})\), \(V_{final} \in \mathbb{R}^{768 + 24}\). The classifier then predicts based on this combined knowledge: \(y_{pred} = \text{Softmax}(W \cdot V_{final} + b)\).

7.1 Limitations of the Hierarchy-Aware RoBERTa Model↩︎

Despite its improved performance, the proposed Hierarchy-Aware RoBERTa model has several limitations. First, the architecture assumes parent-level metadata availability during inference, though this is consistent with coarse-to-fine vulnerability pipelines where high-level categories are readily identifiable via static analysis tools (SAST) or manual triage. Second, the hierarchy is treated as fixed, and the model may not generalize to new or rearranged nodes without retraining, as the CWE taxonomy is subject to continual updates. Third, only direct parent information is captured, disregarding multi-hop ancestors, sibling relationships, and tree depth, which may limit discrimination among deeply nested classes such as Compound and Pillar. When training data is virtually absent, hierarchy awareness alone is insufficient, suggesting that weak supervision or external knowledge may be necessary. Finally, the learnable parent embeddings add modest parameter overhead that could hinder scaling to large taxonomies, and the approach has been evaluated only within the CWE domain, leaving generalizability to other hierarchical tasks empirically unvalidated.

8 Conclusion↩︎

This study examined the effectiveness of oversampling techniques and hierarchy-aware modeling to address class imbalance in CWE vulnerability classification. Through extensive experimentation, we demonstrated that commonly used oversampling methods, such as SMOTE and ADASYN, provide limited benefits for machine learning models (i.e., RF and SVM) and consistently degrade performance for deep learning models (i.e., CNN and BiGRU). These methods generate synthetic samples through linear interpolation in embedding space, an assumption that does not hold for contextualized text representations and often disrupts the semantic and hierarchical integrity of CWE labels.

To overcome these limitations, we proposed a Hierarchy-Aware RoBERTa model that explicitly integrates parent–child relationships from the CWE taxonomy into the classification process. By incorporating learnable parent embeddings alongside contextual language representations, the model enforces structural consistency and improves discrimination among minority and semantically dependent classes. The proposed model achieves a weighted F1-score of \(0.76\) without data augmentation, outperforming all baselines, including BERT, which achieved \(0.74\), demonstrating that structural priors are more effective than synthetic data generation in hierarchical imbalanced cybersecurity datasets.

Despite these gains, extremely sparse categories such as Compound and Pillar remain difficult to predict, highlighting the inherent limitations of supervised learning when training data is severely limited. Future research may extend the hierarchy-aware framework to incorporate multi-hop ancestry and graph neural networks over the full CWE hierarchy, explore weak supervision and label propagation for extremely sparse classes, and evaluate the proposed approach across other hierarchical cybersecurity datasets to establish its generalizability beyond the CWE domain.

Acknowledgment↩︎

This research is partially supported by the U.S. National Science Foundation (Award #: 2319802).

References↩︎