1 Introduction↩︎

Society has become more dependent on digital systems due to, e.g., smart cities [1], Internet of Things [2] and Artificial Intelligence (AI) [3]. This has resulted in a corresponding surge in sensitive data, an increasing demand for the development and deployment of digital devices, and a persistent need for user education and training. A fundamental pillar in sustaining the digital ecosystem is cybersecurity. Cybersecurity professionals act in protecting against cyber-attacks [4], assessing cyber risks [5], safeguarding user privacy [6], protecting minors in online environments [7], and fostering user trust [6]. Despite its critical importance, the estimated cybersecurity workforce gap was at 4.8 million globally in 2024 [8].

At the same time, educational programs struggle to keep up with the fast-paced changes in the cybersecurity landscape [9]–[12]. This dynamic landscape, coupled with the inherently multidisciplinary nature of the field, underscores a growing demand for personalized learning paths [13]. Personalized curricula enable students to either specialize in their area of interest or gain broad knowledge across multiple disciplines. Nevertheless, several challenges persist:

• There are large differences between institutions regarding which components of cybersecurity education should be classified as ‘core’ or ‘elective’ [13]. Core curricula might lack essential topics relevant to industry.

• Increasing responsibility is placed on the student to select ‘the right’ courses for their future career. However, research shows that students primarily tend to choose electives based on their perceived difficulty, instead of the student’s envisioned career [14].

• Adequate guidance counseling should be available to advise students on their curriculum [15], which requires significant resources.

• Ensuring that curricula remain aligned with recent developments and industry demands presents a significant challenge for educators and curriculum developers [9].

To address these challenges, we propose CurricuLLM, a novel end-to-end framework for cybersecurity curriculum analysis based on fine-tuned Large Language Models (LLMs).¹ LLMs excel at processing text, making them particularly suitable for analyzing and interpreting educational guidelines and course material. Prior work has demonstrated the proficiency of these models in processing disciplinary cybersecurity knowledge [17], [18].

By harnessing the language comprehension capabilities of LLMs, CurricuLLM automates the assessment of cybersecurity curricula against standardized educational and workforce frameworks. While our current implementation utilizes the Cybersecurity Curricula 2017 (CSEC2017) [19] and the 2025 National Initiative for Cybersecurity Education (NICE) workforce framework [20] as illustrative examples, CurricuLLM is designed to flexibly support other competency or knowledge frameworks. That is, our CurricuLLM pipeline is built to be easily adapted to other workforce or curriculum frameworks with similar structure, even if the domain is unrelated to cybersecurity.

CurricuLLM enables mapping knowledge requirements of cybersecurity job roles (defined in the NICE framework) to corresponding Knowledge Areas (defined in the CSEC2017 framework), assisting educators, guidance counselors and students in identifying knowledge gaps, aligning learning outcomes with evolving skill demands, and tailoring the students’ skill set towards a desired career path. Ultimately, CurricuLLM contributes to more effective and targeted cybersecurity education.

Technically, CurricuLLM comprises two Language Models: PreprocessLM, which is used to convert the diverse fine-tuning and evaluation datasets into a standardized format suitable for machine learning, and ClassifyLM, which maps course content to one or multiple Knowledge Areas (KAs). These areas were defined in the CSEC2017 framework and signify key competencies covered in cybersecurity education. ClassifyLM is fine-tuned on the NICE workforce framework, which was labeled by [12], and enriched with course-specific knowledge using a synthetic dataset derived from the CSEC2017 framework. Several Natural Language Processing (NLP) models were evaluated as candidates for ClassifyLM. A fine-tuned Bidirectional Encoder Representations from Transformers (BERT) [21] model ultimately outperformed both zero-shot autoregressive models like ChatGPT [23] and traditional machine learning baselines. The KAs extracted from ClassifyLM can then be used to match course content to related job roles, as listed in the NICE framework.

Since the KAs can overlap by design, KA classification is a multi-label task. Furthermore, subjectivity in the definitions of KAs leads to low inter-rater agreement. Therefore, we compare the KAs assigned by CurricuLLM with 3 cybersecurity experts, that have a background in analyzing the CSEC2017 framework, as well as a control group of more than 60 cybersecurity experts inquired at scientific workshops. Our results demonstrate that CurricuLLM can effectively replicate expert judgments, making it the first LLM-based method shown to automate cybersecurity curriculum analysis supported by quantitative and qualitative validation.

To demonstrate the practical application of CurricuLLM, we present an evaluation of three real-world cybersecurity programs using CurricuLLM: the M.Sc. programs in cybersecurity at Kungliga Tekniska Högskolan (KTH) in Sweden [24], Nanyang Technological University (NTU) in Singapore [25] and the M.Sc. in Information Technology-Information Security (MSIT-IS) offered at Carnegie Mellon University (CMU) in the United States [26]. Our analysis examines these programs at varying levels of granularity, ranging from individual courses to a program-level analysis.

To ensure that our methodology reflects the latest workforce requirements, we utilize CurricuLLM to annotate the updated 2025 NICE workforce framework, developed by the National Institute of Standards and Technology [20]. Specifically, CurricuLLM assigns KAs to each Knowledge Description (KD) within the framework, where KDs represent competency requirements defining knowledge that cybersecurity professionals should possess. The extracted KA labels are benchmarked against expert annotations and subsequently used to update the role-based weightings introduced by [12]. This process ensures that CurricuLLM’s mapping between cybersecurity curricula and job roles reflects the most recent competency requirements. Furthermore, we conduct a longitudinal analysis to examine the evolution of these competency demands between 2017 and 2025, and we identify which KAs should be covered in education based on current job market demands. In doing so, this work addresses the persistent cybersecurity workforce skills gap.

Therefore, our key contributions are as follows.

• A Cybersecurity Curriculum Assessment Tool: We introduce a novel LLM-based tool called CurricuLLM, designed to assess the compliance of existing cybersecurity curricula with cybersecurity job market needs. This tool is intended for use by a) curriculum developers aiming to align their programs with evolving industry and workforce demands, and b) students or their guidance counselors seeking to tailor students’ educational pathways to meet the requirements of specific cybersecurity roles or general job market needs.

• Guidelines for LLM-based Curriculum Design: We present a comprehensive methodology for leveraging fine-tuned large language models (LLMs) in cybersecurity curriculum development. This includes dataset preparation strategies, quantitative comparisons of alternative machine learning approaches, and validation on real-world curricula. This approach can easily be adapted to areas unrelated to cybersecurity.

• Workforce-Aligned design: We extend CurricuLLM to integrate the 2025 NICE Workforce Framework, ensuring that its job role-based weighting scheme remains aligned with the most recent definitions of cybersecurity competencies.

This paper is organized as follows. Section [sec:relevant_work] provides background on cybersecurity curriculum design and LLMs. Section [sec:example] presents a case study of personalized curriculum design with CurricuLLM. Section [sec:methods] details our methodology, including dataset preparation, LLM fine-tuning, and computing job role-based weights based on workforce demands. Section [sec:results] reports quantitative and qualitative experimental results. Section [sec:discussion] explores limitations, directions for future research, and alternative applications of CurricuLLM. Finally, Section [sec:conclusion] summarizes our main findings.

2 Background↩︎

In this section, we present background knowledge and state-of-the-art literature on cybersecurity curriculum development (Section 2.1), automated curriculum planning (Section 2.2), and LLMs (Section 2.3).

2.1 Cybersecurity Curriculum Development↩︎

The significance of developing, maintaining, and continuously improving cybersecurity curricula has been widely acknowledged within the research community [27]. Consequently, numerous scholarly works have addressed on this topic from various perspective, focusing on aspects such as methods on enhancing cybersecurity pedagogy [28], [29], workforce training on both national and international levels [12], [30], [31], developing quality criteria for cyber security MOOCs (Massive Open Online Courses) [32], and the identification of essential topics to be included in cybersecurity curricula [33], [34].

This study builds upon the following foundational frameworks.

• The Cybersecurity Curricula 2017 (CSEC2017) framework, developed by the Joint Task Force on Cybersecurity Education (consisting of professionals from ACM, IEEE, AIES, and IFIP) [19]. CSEC2017 organizes key concepts in the cybersecurity domain into eight Knowledge Areas, which are further decomposed into Knowledge Units. A detailed definition of each KA is presented in Table 2.

• The National Initiative for Cybersecurity Education (NICE) workforce framework v2.0.0., published by the National Institute of Standards and Technology [20]. The NICE framework establishes a standardized taxonomy for the cybersecurity workforce, consisting of 5 workforce categories and 41 cybersecurity work roles. Each work role is decomposed into detailed Knowledge, Skill and Ability descriptions.

• The work of [12], which provides a mapping between the NICE and CSEC2017 framework. Their methodology involved manually classifying NICE Knowledge Description into the eight CSEC2017 Knowledge Areas. This process revealed a range of topics relevant to cybersecurity education but not covered by CSEC2017. To address this, they proposed a new, ninth Knowledge Area ‘Miscellaneous’, which includes topics such as Computer Science and Information Technology. Note that this work was based on an earlier version of the NICE framework (v1.0.0. [35]). For an overview of the changes between the two versions of the NICE framework, we refer the reader to Section 5.3.2.

2.2 Automated Curriculum Planning↩︎

The first reported use of AI for curriculum planning was by [36]. However, this concept received limited attention until recent years, when AI began to play a more prominent role in educational applications such as formulating learning objectives [37], intelligent tutoring systems, and automated assessment [38].

Classical approaches to automate curriculum design include heuristic-based methods, such as course dependency graphs [39] and genetic algorithms [40]. More recently, deep learning frameworks employing Long Short Term Memory (LSTM) networks, a type of Recurrent Neural Networks, have been applied to curriculum design [41], [42]. The advent of Large Language Models (LLMs) has further advanced the field by enabling fully automated curriculum design, with [42] representing the first work to apply LLMs specifically to cybersecurity curriculum development. This work serves as a key baseline for CurricuLLM.

2.3 LLMs↩︎

Natural Language Processing (NLP) focuses on the machine learning-based processing of textual data. Since neural networks are designed to process numerical inputs rather than text, all NLP methods first convert textual data into numerical representations through a process known as tokenization. Historically, this was achieved using frequency-based feature extraction techniques such as bag-of-words (BoW) [43] or frequency-inverse document frequency (TF-IDF) [44], which assign a number to each unique word in a document based on its frequency of occurrence. Nowadays, deep learning methods for NLP or so-called Large Language Models (LLMs) have revolutionized the field of NLP. These models map words to numerical tokens via training on large-scale text corpora [21]. Unlike traditional NLP methods that break up sentences into unordered sets of words, LLMs preserve and leverage the sentence structure and context.

LLMs can be categorized into autoencoding and autoregressive models based on their architectural structure. Autoencoding models, such as Bidirectional Encoder Representations from Transformers (BERT), are trained using a denoising objective where inputs are masked and an encoder-decoder model is optimized to recover the unmasked input. BERT uses bidirectional self-attention to incorporate context from both preceding and subsequent words within a sentence. In contrast, autoregressive models such as the GPT models (including ChatGPT [23]), are trained to predict the next word in a sentence given the preceding context.

While off-the-shelf LLMs perform well on general-purpose tasks, domain adaptation is often necessary for specialized applications, such as curriculum design. In the case of autoencoding models, this can be done by fine-tuning the parameters of an off-the-shelf model on a small domain-specific dataset. Autoregressive models, in contrast, are typically adapted through few-shot learning on a limited number of examples, allowing modification of the model’s behavior without explicitly altering the underlying parameters [45]. In this study, we fine-tune BERT [21], an autoencoding model, that comprises approximately 110 million parameters. BERT is particularly suitable for text classification tasks [46], e.g. classifying course content into one of the nine KAs (eight KAs from CSEC2017 and one KA from [12]). Additionally, our training dataset is relatively small, which makes fine-tuning larger models infeasible in practice due to both the substantial computational cost and the increased risk of overfitting when adapting models with billions of parameters. For context, ChatGPT-4o mini is estimated to contain around 8 billion parameters [47], highlighting the significant computational disparity between the two approaches.

To date, [42] present the only other LLM-assisted curriculum design tool. Notably, their methodology differs fundamentally from ours: although they utilize the BERT tokenizer, they substitute the subsequent BERT base and classification model with a Long Short-Term Memory (LSTM) model. Furthermore, their model is fine-tuned based on job advertisements, whereas our approach explicitly fine-tunes on educational standards and maps curricula to workforce requirements. In our fine-tuning methodology (Section 4.2 and 8), we adopt the methodology proposed by [42] as a baseline to evaluate the performance of our approach.

3 Case Example: Personalized Curriculum Design using CurricuLLM↩︎

To illustrate how CurricuLLM can be used for personalized curriculum design, we analyze the following case example: Alice is a Master’s student in cybersecurity at KTH. They want to become a vulnerability analyst. Which elective courses should they take to best prepare themselves for this role?

We address this problem methodologically, as illustrated in Figure 1. Alice must select 4 out of the 12 elective courses listed on the top left of Figure 1. In this work, we adopt the Knowledge Areas, as proposed in the CSEC2017 curriculum [19], to compare academic courses with job market demands. A brief description of all KAs is provided in Table 2. Each KA encompasses an essential component of cybersecurity education, with the exception of the ‘Miscellaneous’ KA, which covers topics previously not included in the CSEC2017 KAs [12]. The color-coding scheme used throughout this paper is also highlighted on the top right of Figure 1.

For each mandatory and elective course in the KTH MSc in Cybersecurity we apply CurricuLLM to compute the distribution of KAs covered in the course. A representative example, the Building Networks System Security elective course, is depicted at the bottom of Figure 1. The KA labeling process follows three stages:

1. Course preprocessing We extract the course title and course description, which are then fed through an instruction-tuned Large Language Model (PreprocessLM) to generate concise topic descriptors that capture the core content of the course.

2. KA classification Each extracted topic is subsequently passed through a multi-label classification Language Model (ClassifyLM). ClassifyLM was fine-tuned on cybersecurity topics derived from the CSEC2017 curriculum and Knowledge Descriptions extracted from the NICE workforce framework, to be able handle both course and workforce data.

3. Label aggregation Finally, all KA labels extracted from the course title and description are aggregated, normalized, and visualized as a pie chart.

The Building Networked System Security course focuses on training students to handle contemporary security problems through problem-based learning and teamwork. As expected, the extracted topic ‘handling contemporary security problems for networked systems’ was assigned KA 4, Connection Security. The aggregated KA labels show that Organizational Security dominates, and System Security, Connection Security, and Miscellaneous account for the remainder of the course content.

This procedure is systematically applied to all courses in the program (see the pie charts for each elective in Figure 1). The aggregated KA distribution for the mandatory component of the Cybersecurity master at KTH is displayed in the center top of Figure 1. Simultaneously, we map the KA profile for the target role of vulnerability analyst based on the mappings provided in Figure 9. When comparing the KA distribution of Alice’s mandatory courses and the KA distribution of the vulnerability analyst job role, we see that Alice primarily lacks courses related to Connection Security (in red) and Miscellaneous (in lime). To identify the most suitable set of four elective courses to address this knowledge gap, we formulate a constrained optimization problem (for more details on this formulation, see 9). The objective is to minimize the divergence between the combined KA distribution of the selected curriculum (mandatory & selected electives) and that of the target job role.

In Alice’s case, the optimization identifies four electives that most closely align their curriculum with the desired role of vulnerability analyst (see the middle-right of Figure 1). The Networked Systems Security course and the Advanced Networked Systems Security course contain the largest proportion of content related to Connection Security, while Building Networked Systems Security includes the largest proportion of Organizational Security out of all elective courses. The resulting final program aligns more closely to the required KA distribution for the Vulnerability Analyst job role than the stand-alone mandatory program. Therefore, Alice’s electives have been chosen in such a way that they increase the correspondence of their curriculum with the requirements from the job market.

4 Methodology↩︎

In this work, we present CurricuLLM, a novel end-to-end framework that leverages LLMs for the analysis of cybersecurity curricula. CurricuLLM consists of the following steps: data preprocessing, KA annotation, and the computation of job role-based weights based on demands in the cybersecurity workforce. The remainder of this section provides a comprehensive description of each component, beginning with data selection and pre-processing (Section [sec:datasets]), followed by the fine-tuning procedure for ClassifyLM (Section [sec:LL2_finetuning]), and the computation of job role-based weights (Section [sec:role-basedweight]).

4.1 Datasets↩︎

Table 3 presents an overview of all datasets used to fine-tune and evaluate CurricuLLM, along with their respective preprocessing methods. The NICE Workforce Framework defines standardized competency specifications for cybersecurity professionals, called Knowledge Descriptions. The KDs contained in NICE framework v1.0.0. were annotated with Knowledge Area labels by [12] and can therefore be used to fine-tune the classification model, ClassifyLM. Because the NICE framework focuses on workforce roles rather than educational content, we complement it with course-based data for fine-tuning and evaluation to ensure that the resulting methodology is well aligned with curriculum analysis. This dataset is synthetic and derived from the CSEC2017 framework. This framework comprises nine Knowledge Areas (including the recently proposed Miscellaneous KA [12]), which are further subdivided into ‘Knowledge Units’ (KUs) containing detailed descriptions of cybersecurity topics covered in the KU. PreprocessLM was used to extract fine-grained topics from these descriptions (as shown in Figure 2). The resulting synthetic dataset contains approximately 2,100 labeled topics.

For our evaluation pipeline, we curated a set of real-world course descriptions collected from the official MSc Cybersecurity curricula at the KTH royal academy of Stockholm, Nanyang Technical University of Technology (NTU) in Singapore and Carnegie Mellon University in Pittsburgh. The courses contained in these curricula provide a diverse and realistic benchmark for evaluating to what extent CurricuLLM generalizes to real academic curricula. Furthermore, NIST has recently released an updated version of the NICE framework, in which only 15 Knowledge Descriptions remain unchanged from the previous release [20]. This framework therefore also used to evaluate our methodology.

LLMs are known to be sensitive to stylistic features such as the prompt length [48], uppercase characters [49], and the formatting of prompts as plaintext, markdown, YAML or JSON [50]. Therefore, our preprocessing pipeline is designed to uniformize the three datasets in Table 3. We convert all data into a standardized format consisting of a single sentence that describes one cybersecurity topic, devoid of unnecessary verbosity. For the NIST KDs, this entailed removing the redundant prefix “Knowledge of”, while the other datasets were processed more rigorously with PreprocessLM. As illustrated in Table 3, the datasets are more homogeneous after pre-processing: All entries are short, in lower-case, and redundant text has been removed. This is particularly noticeable for the CSEC2017 synthetic and KTH, NTU and CMU curricula data. The data standardization process ensures that both fine-tuning and evaluation data follow similar formatting structures, enhancing model robustness.

None

4.2 ClassifyLM fine-tuning↩︎

We evaluated various different modeling strategies to effectively classify course descriptions into one or multiple Knowledge Areas, ranging from traditional machine learning methods to fine-tuned LLMs. The fine-tuned BERT model [21] emerged as the best candidate to serve as ClassifyLM, due to its superior KA classification performance on the fine-tuning dataset (described in Section 4.1), surpassing traditional machine learning techniques, the baseline developed by [42], and zero-shot prompting of ChatGPT-4o-mini [23] and DeepSeek-V3 [51]. The full fine-tuning and model selection methodology is outlined in 8. The final ClassifyLM model was trained using the following settings:

• Model: BERT (listed as bert-base-uncased in the HuggingFace library)

• Loss Function: Binary Cross-Entropy

• Optimizer: Adam

• Batch Size: 64

• Learning rate: \(4\times 10^{-5}\)

• Epochs: 10

These hyperparameters were selected based on cross-validated macro-F1 scores. All code containing baselines, our implementation of CurricuLLM, data preprocessing and evaluation methodologies has been included on our publicly available GitHub [52].

4.3 Cybersecurity job role-based weights↩︎

In this work, we adopt the Knowledge Areas as a common frame of reference between cybersecurity curricula and workforce competencies. While PreprocessLM and ClassifyLM are used to map courses to KAs, we build on the methodology described by [12] to compute the relative importance of the KAs for the overall job market, work role categories, and specific work roles. The overall procedure is visualized for the ‘Vulnerability Analysis’ job role in Figure [fig:nice_framework], but the results for all NICE job roles can be found in 12.

The NICE cybersecurity workforce framework [20] is utilized to derive role-based weights for either individual job roles, job categories, or the overall cybersecurity labor market. As shown in Figure [fig:nice_framework], the framework is organized into five overarching work role categories. Each of the 41 job roles is linked to exactly one category.

Each job role in the NICE framework is characterized by three types of competency descriptors: Knowledge Descriptions, Skill Descriptions, and Task Descriptions. This study focuses exclusively on KDs, as these are directly relevant for curriculum analysis. KDs are standardized descriptions of knowledge, e.g. “Knowledge of penetration testing tools and techniques", that are often applicable to multiple job roles.

Most KDs were manually labeled with corresponding KAs by [12]. However, an updated version of the NICE framework released in 2025 introduces significantly reformulated KDs, rendering many of the previous annotations obsolete. To ensure our role-based weightings accurately reflect the current standard, we adopt the most recent version of the NICE framework and automatically label all newly defined KDs using ClassifyLM.

This comprehensive mapping allows us to compute a target KA distribution for any given job role. The calculation involves aggregating the KA distribution vectors of all KDs associated with a specific role and normalizing the resulting vector such that it sums to 100%. Figure [fig:nice_framework] provides a visual overview of this process and presents the final normalized KA distribution as a pie chart. Note that it is also possible to compute the desired KA weighting for a job category by averaging the KA distributions of all roles contained in this category (see Section 5.3.2). Similarly, the aggregate KA distribution for the overall cybersecurity professional can be estimated by computing the mean KA distribution across all job categories, weighted by their respective labor market demand (see Section 5.3.1).

5 Validation Results↩︎

This section presents a performance analysis of CurricuLLM. A quantitative comparison between CurricuLLM and human annotators is detailed in Section [sec:quant_cllm_humans]. Subsequently, Section [sec:qual_analysis] provides a qualitative evaluation, encompassing both a single-course case study and an analysis of three real-world curricula. Finally, Section [sec:future_proof] aligns CurricuLLM to workforce needs by estimating the job market relevance of each Knowledge Area and the desired KA distribution for the NICE work role categories.

5.1 Quantitative comparison: CurricuLLM vs Human Annotators↩︎

Since CurricuLLM was developed to automate manual curriculum design, it is important to compare its performance to human experts. To this end, we conducted experiments involving three distinct groups of domain specialists, differing in their familiarity with the CSEC2017 framework and the type of data they annotated.

• Expert Group X comprises three domain experts, denoted as \(X_1\), \(X_2\), and \(X_3\), who were familiar with the CSEC2017 framework prior to the study. These experts annotated all data analyzed in this section.

• Control Group A consisted of 35 cybersecurity experts inquired to label real-world course data during the Seminar of Swedish IT Security (SWITS) 2025 in Stockholm, Sweden. Each participant labeled up to seven extracted course topics.

• Control Group B consisted of 34 experts consulted during the Secure Systems Demo Day 2025 in Espoo, Finland. Each participant in this group labeled 3 KDs as defined in the NIST 2025 framework.

The participants in Control Groups A & B were provided with concise definitions of each Knowledge Area (as shown in Table 2) and were asked to assign one or more KAs to a small subset of either real-world curriculum data or KDs. Section 5.1.1 presents the results of curriculum analysis, using Expert Group X and Control Group A as reference points for comparison with CurricuLLM. Section 5.1.2 reports the results of labeling NIST KDs, evaluated with Expert Group X and Control Group B as a reference.

5.1.1 Curriculum analysis of KTH, NTU, CMU↩︎

The level of agreement between CurricuLLM and human experts in classifying course data was evaluated as follows. First, a sample of nine courses was selected comprising three courses randomly chosen from each of the KTH, NTU, and CMU curricula. Next, PreprocessLM extracted 79 unique topics from these course descriptions, which served as input for both the ClassifyLM model and human annotators.

The aggregated results of this comparative analysis are depicted in Figure [fig:combined_analysis]. The raw data, i.e. the annotations of all 79 extracted topics by Expert Group X, Control Group A, and CurricuLLM, are listed in Table [tab:network_systems_security] and 10. Here, Figure [fig:curriculum_analysisv2] illustrates the percentage-wise agreement between all pair-wise combinations of annotators, where agreement was defined as the assignment of at least one common KA label to a given topic. For example, if one annotator assigned [1,4] and another assigned [2,4] to the same topic, the pair was considered to be in agreement.

In addition, Figure [fig:curriculum_analysis] reports the corresponding Cohen’s kappa, which quantifies pair-wise agreement beyond chance. Cohen’s kappa ranges from -1 to 1, where 0 indicates agreement equivalent to chance, and 1 signifies perfect agreement. This metric is stricter than percentage-wise agreement, as it requires complete consensus for a score of 1. The inter-annotator agreement was relatively high among Expert Group X, with the percentage-wise agreement ranging from 71–87%, and Cohen’s kappa values between 0.38 – 0.53. As illustrated in Figure [fig:curriculum_analysisv2], it appears that the average agreement of CurricuLLM with Expert Group X (61%) was comparable to agreement of Control Group A and Expert Group X (66%). However, the results in Figure [fig:curriculum_analysis] reveal a different trend: the mean Cohen’s kappa compared to Expert Group X was significantly higher for CurricuLLM (\(\kappa\) = 0.35) than for Control Group A (\(\kappa\) = 0.26). This discrepancy indicates that, although participants in Control Group A tended to assign a greater number of KA labels per topic (increasing the chance of overlap), their classifications were less aligned with the CSEC2017 curriculum than those produced by CurricuLLM. This finding suggests that the KA labeling task requires familiarity with the CSEC2017 curriculum: while Control Group A participants were provided with basic definitions of each KA, this information alone was insufficient to achieve expert-level consistency with members of Expert Group X.

Furthermore, the Cohen’s kappa results indicate that CurricuLLM could automate course labeling with only a minor loss in performance relative to human experts. The mean agreement between CurricuLLM and Expert Group X (\(\kappa\) = 0.35) is comparable to the agreement observed between Expert Group X annotators \(X_2\) and \(X_3\) (\(\kappa\) = 0.38). In summary, CurricuLLM substantially outperformed Control Group A, consisting of cybersecurity experts previously unaware of the CSEC2017 framework, with only a marginal loss of performance compared to the CSEC2017-aware experts in Expert Group X (\(X_1\), \(X_2\), and \(X_3\)).

5.1.2 NIST KD label analysis↩︎

To provide up-to-date workforce requirements and to assess the generalization capabilities of CurricuLLM beyond academic curricula, we conducted a second experiment focusing on the Knowledge Descriptions defined in the updated 2025 NICE Framework. This dataset differs fundamentally from course data, as each KD represents a specific cybersecurity skill or concept associated with professional competency requirements. As discussed in Section [sec:role-basedweight], the KDs form the foundation for computing the required KA distribution from the perspective of the cybersecurity job market, both for individual job roles and for the overarching workforce.

To evaluate the performance of CurricuLLM on KD data, we analyzed 50 randomly selected KDs from the updated 2025 NICE Framework, ensuring that none overlapped with the entries used by [12]. Each KD was preprocessed by removing the non-discriminative prefix ‘knowledge of’ before classification. Subsequently, ClassifyLM was used to map each KD to one or more corresponding Knowledge Areas.

The results are summarized in Figure [fig:combined_kd_analysis]. Figure [fig:comparison_experts_swits_gpt_4o_mini] presents the percentage of overlapping KA assignments across annotator pairs, representing the proportion of KDs that shared at least one common KA label. Figure [fig:curriculum_analysis] reports the inter-rater agreement measured using Cohen’s kappa (\(\kappa\)), which accounts for agreement by chance. Note that a subset of 15 KDs was selected for comparative analysis with Control Group B (indicated as \(*\) in Figure [fig:combined_kd_analysis]). Overall, both agreement metrics were considerably lower for KD data than for course data, with the mean percentage-wise agreement decreasing from 67% to 56%, and the mean Cohen’s kappa dropping from 0.34 to 0.29. This outcome is expected, as the CSEC2017 framework was primarily designed to label courses, whereas KDs represent occupational requirements. Nonetheless, while the absolute values declined, the relative trends remain consistent across datasets.

Among Expert Group X, the mean percentage-wise agreement was 54% and the mean Cohen’s kappa was \(\kappa = 0.35\). The average agreement between CurricuLLM and Expert Group X was 57% and \(\kappa = 0.35\), effectively matching the mean inter-annotator reliability among the human experts. In contrast, the agreement between Control Group B and Expert Group X was substantially lower (\(\kappa = 0.20\)). This pattern mirrors the findings from Section [sec:curr_results_quant], where CurricuLLM also outperformed human control participants unfamiliar with the CSEC2017 framework.

Taken together, these results demonstrate that CurricuLLM maintains expert-level consistency even when applied to non-academic, workforce-oriented data.

5.2 Curriculum Analysis↩︎

To demonstrate the practical utility of CurricuLLM for curriculum design and analysis, we present a qualitative evaluation. This evaluation illustrates CurricuLLM’s functionality at the granular level of a single course (Section [sec:single_course]) and provides a comparative assessment of entire curricula from three institutions (Section [sec:whole_curriculum]).

5.2.1 Single Course analysis: Building Networked Systems Security↩︎

CurricuLLM was designed to analyze individual courses in a modular fashion, such that an overall curriculum can be built by selecting the courses that approximate the KA distribution required by the workforce. To illustrate this process, we examine a single course case study: the Building Networked Systems Security course offered at KTH (c.f. the case example presented in Section [sec:example]). Note that similar course-level data is available for 8 other courses, as outlined in 10.

Our analysis focuses on publicly available course descriptions, as this information is available for most cybersecurity programs, ensuring CurricuLLM can be applied widely. Course descriptions, however, vary significantly in style. In some countries, such as Sweden, they are legally binding documents, whereas in other countries, course descriptions primarily serve as promotional material aimed at recruiting prospective students. Therefore, to be able to be processed further by a LLM, the course descriptions need to be standardized. As explained in Section [sec:datasets], the descriptions are passed through PreprocessLM to extract key pedagogical topics.

Following this procedure, the Building Networked Systems Security course description was processed by PreprocessLM, resulting in eight distinct pedagogical topics (listed in the leftmost column of Table [tab:network_systems_security]). These topics were subsequently annotated by three groups: (1) 35 participants from the SWITS conference (Control Group A), (2) our panel of curriculum-aware experts \(X_1\), \(X_2\), and \(X_3\) (Expert Group X), and (3) the CurricuLLM model. In general, Control Group A exhibits little pair-wise overlap between experts #1, #2, and #3. Consistent with the broader findings in Section [sec:curr_results_quant], participants in Control Group A had a tendency to assign more and less accurate labels to each topic, resulting in a relatively high percentage-wise agreement with Expert Group X but lower Cohen’s kappa scores.

As displayed in Table [tab:network_systems_security], among Expert Group X, consensus was achieved only on the topic Problem-based learning, which all assigned to the Miscellaneous KA. Conversely, lower agreement was observed for more applied topics with weaker ties to cybersecurity core knowledge, such as ‘Critically assessing the efficiency of alternative solutions’. These discrepancies highlight the inherent subjectivity of the KA annotation process and suggest that some topics admit multiple valid interpretations.

CurricuLLM showed perfect alignment with all experts on ‘Problem-based learning’, agreement with Expert \(X_1\) on ‘Handling contemporary security problems for networked systems’, with Expert \(X_2\) on ‘Investigating requirements for networked systems security’ and partial alignment across multiple experts on ‘Teamwork in cybersecurity’. However, for topics more tangentially related to cybersecurity, such as ‘Preparing solutions with professional tools’ expert diverged or opted for the Miscellaneous KA, while CurricuLLM assigned them to Organizational Security. This resulted in a larger representation of Organizational Security as compared to Expert \(X_1\), \(X_2\), and \(X_3\). This discrepancy indicates that the experts utilize a different definition of Miscellaneous, i.e. ‘all competencies that are not directly cybersecurity related’, while CurricuLLM is directly trained on the CSEC2017 description and the miscellaneous class. For example, ‘Critically assessing the efficiency of alternative solutions’ overlaps with the operational and tactical management Knowledge Unit contained in Organizational Security and little to no overlap with the Knowledge Units contained in Miscellaneous (see Table 2). However, this topic could also be considered outside the scope of cybersecurity and therefore be assigned to Miscellaneous, even though it does not relate to Knowledge Units contained in this KA.

Then, moving towards curriculum design, the expert KA labels for the Building Networked Systems Security course are aggregated by taking the sum of all assigned labels to the topics contained in the course and normalizing the result to produce a probability distribution. The resulting normalized KA distributions are visualized as pie charts, as displayed in Figure [fig:Network_Security_course]. As expected, the annotations from Control Group A approximate a uniform distribution over all KAs. Experts \(X_1\) and \(X_3\) assign the course the most to Miscellaneous and Connection Security, whereas expert \(X_2\) favors System Security. Each of these annotations are defensible, since the course involves aspects of networks, systems, and also relates to softer skills that could be considered Miscellaneous. CurricuLLM assigns most topics to Organizational Security, and assigns the rest to Miscellaneous, Connection and System Security, in accordance with the experts in Expert Group X.

5.2.2 Entire curriculum analysis: KTH, NTU and CMU↩︎

CurricuLLM facilitates a fully automatic approach to curriculum analysis. Although CurricuLLM was initially conceived to analyze single courses and compose a curriculum based on these courses, it is equally capable of analyzing an entire curriculum by aggregating the KA labels of all courses contained in the curriculum. This is relevant to analyze to what extent a core curriculum covers job market demands, or whether provided conditionally elective courses are diverse enough for students to develop specialist competencies.

To highlight the value of automated curriculum assessment, we analyzed three programs: the KTH MSc in Cybersecurity [24], the NTU MSc in Cybersecurity [25], and the CMU MSc in Information Technology-Information Security (MSIT-IS) [26]. The results of this comparative analysis are presented in Figure [fig:entire_curriculum]. For the KTH and NTU MSc, the results have been split into core and elective, where core indicates mandatory courses and elective indicates an aggregate of all conditionally (i.e. not free choice) elective courses, excluding degree projects. The CMU MSIT-IS degree’s core consists mostly of conditionally elective courses. For the analysis provided here, a representative sample program fulfilling the degree requirements was selected.

A preliminary observation is that Organizational Security takes up a large proportion of all analyzed curricula. This is consistent with the single-course analysis provided in Section [sec:single_course]. Additionally, all three programs show a relatively light coverage of Component Security and Human Security, indicating a common curricular trend that prioritizes systemic and organizational aspects of cybersecurity over content related to hardware and human security.

KTH’s core curriculum consists of 7 courses that together contribute to almost a third of the program: Theory and Methodology of Science, Theory of Science and Scientific methods in Cybersecurity, The Cybersecurity Engineer’s Role in Society, Cybersecurity Overview, Cybersecurity in a Socio-Technical Context, Applied Cryptography, and Ethical Hacking. As depicted in Figure [fig:entire_curriculum], this results in a larger contribution of Organizational Security, Societal Security, Human Security and System Security in the program. In contrast, the elective courses place greater emphasis on Connection Security and Software Security, with courses like ‘Building Networked Systems Security’ and ‘Language-based Security’.

NTU’s core curriculum consists of 4 courses that contribute to 40% of the curriculum; Computer Security, Application Security, Cryptography, Security and Risks Management. As compared to the core of KTH, this program has a stronger focus on Human Security and Data Security, while contributing less Societal Security. The 9 conditionally elective courses offered at NTU deliver a diverse palette of System Security courses (like Cyber physical systems security) and Connection Security (e.g. Network security).

The analyzed CMU sample core program (consisting of Fundamentals of Telecommunications and Computer Networks, Browser Security, Introduction to Information Security, Information Security Risk Management, Information Security Policy and Management and Academic & Professional Development) contributes to approximately 40 percent of the MSIT-IS program. Compared to the other programs, it shows less emphasis on technical areas like Data Security, System Security and Software Security, and more focus on Miscellaneous topics and Organizational Security.

5.3 Results related to Job Market↩︎

To assess how cybersecurity curricula can be aligned with current or emerging job market needs, we analyzed the correspondence between the KAs from the CSEC2017 framework and the cybersecurity work roles defined in the latest NICE 2025 framework [20]. Specifically, this section examines (i) the overall relevance of each KA to the job market (Section [sec:KA_to_job]) and (ii) the desired KA distribution for each NICE job category (Section [sec:job_to_KA]). Together, these analyses provide insight in how curricula can be adapted to address the cybersecurity workforce gap.

5.3.1 Job Market Relevance of Knowledge Areas↩︎

To be able to design curricula targeted towards gaps in the cybersecurity workforce, it is important to know which aspects of the cybersecurity curriculum are most relevant for the job market. To this end, we extracted job demand data from the CyberSeek platform [53], which reports the number of job openings per NICE Cybersecurity Workforce job role in the US between May 2023 and April 2024. For each NICE job role, the demand was interpreted as a weight over the total number of listings. As explained in Section [sec:role-basedweight] and illustrated in Figure [fig:nice_framework], each NICE work role can be decomposed into Knowledge Descriptions, which are then mapped to Knowledge Areas contained in the CSEC2017 framework using the CurricuLLM framework. By propagating CyberSeek job demand weights through this mapping, we derived the aggregated relevance for each KA across all work roles, as depicted in Table 8. It should be noted that the recently introduced Operational Technology Cybersecurity Engineering role, part of the Design & Development category in NICE 2025, had no available job demand data at the time of this analysis.

Data in Table 8 reveal that Organizational Security forms the most dominant KA. This can in part be attributed to lexical patterns used in the revised KDs, e.g., the occurrence of words like ‘project’ or ‘management’, which tend to trigger classification under Organizational Security by CurricuLLM. This finding aligns with prior manual labeling efforts of KDs [12], which similarly identified Organizational Security as the most important KA. This consistency suggests that the trend reflects a genuine emphasis in the labor market rather than a model artifact. Other high-weight KAs include Miscellaneous, Connection Security, Data Security, and Societal Security. The relatively large share of Miscellaneous is particularly noteworthy, since it indicates the need for cybersecurity experts with multidisciplinary knowledge.

To clarify what conclusions can be drawn from Table 8, we note that the data extracted from CyberSeek should be understood as describing market demand rather than the underlying normative needs of the cybersecurity ecosystem. Using the number of job listings as a measure of demand represents only one possible way to quantify workforce needs. Alternative options, such as weighting by the number of distinct organizations seeking a given role, could yield different demand distributions and highlight different KAs. Our analysis therefore solely estimates which Knowledge Areas best align with vacancies. From an information-theoretic perspective, an optimal match between an individual’s skill set and job market demands would therefore correspond to a probability distribution over all KAs that approximates the empirical distribution derived in Table 8, for instance measured through the Kullback–Leibler divergence [54].

5.3.2 Desired KA distribution per NICE Job Category↩︎

Beyond the aggregate market relevance of individual KAs, it is valuable to understand the specific KA profile required for different career paths. This information supports several use cases, such as identifying which elective courses best align with a student’s desired career, or determining which NICE job categories a given curriculum prepares students most effectively for.

Therefore, we analyzed the desired KA distributions for every NICE job category using the methodology described in Section 4.3. The results for each NICE job role can be found in 12. To enable a longitudinal comparison of the requirements per NICE job category, we mapped the categories from NICE 2017 (Framework Components v1.0.0. [35]) to their counterparts in NICE 2025 (Framework Components v2.0.0. [20]). The largest difference between v1.0.0. and v2.0.0. was the removal of the ‘Collect & Operate’ and ‘Analyze’ categories, which were taken up into the DoD Cyber Workforce Framework (DCWF) [22]. Therefore, we discarded these categories from our analysis.

Figure 9 depicts the resulting KA distributions for the NICE job categories Oversight & Governance (OG), Design & Development (DD), Implementation & Operation (IO), Protection & Defense (PD), and Investigation (IN) for both 2017 (top) and 2025 (bottom). Several important distinctions between the 2017 and 2025 desired KA distribution can be noted. Firstly, Organizational Security is more prominent across all categories, reaching at least 28% in each. At the same time, the contribution of Miscellaneous has shrunk significantly. As discussed in Section 5.2, this shift can be attributed to differences in labeling behaviour between human annotators and CurricuLLM: whereas human experts assigned topics they consider to be outside of the scope of cybersecurity to Miscellaneous, CurricuLLM tends to assign these to Organizational Security.

Furthermore, Figure 9 indicates that the distribution of KAs has become more uniform over time. In the NICE 2017 framework, the importance of KAs varied widely across workforce categories. For instance, Component Security contributed 9% to OG, while it was considerably less prominent in other categories. In the 2025 Framework, the contribution of Component Security has equilibrated to a narrow range of 1% and 4% across all categories. Similarly, Connection Security took up between 12% and 33% in 2017 and between 9% and 16% in 2025. However, as can be observed in 12, the distributions for individual job roles still vary significantly.

The largest changes are observable for the PD and IN workforce categories. Coincidentally, these categories have been modified most vigorously between 2017 and 2025. PD has absorbed several redefined work roles, including some formerly categorized under ‘Investigate’. At the same time, IN was reduced from three to two work roles. In contrast, IO, DD, and OG only changed marginally between 2017 and 2025. This indicates that the reported differences in KA distributions could in part be attributed to structural changes in the NICE framework. Additionally, the mean length of individual KDs has decreased from approximately 85 characters in 2017 to 52 characters in 2025, indicating a significant change to the phrasing of Knowledge Descriptions between 2017 and 2025.

6 Discussion↩︎

Our results demonstrate the value of CurricuLLM as an end-to-end cybersecurity curriculum assessment tool. We contextualize these findings by outlining future research directions (Section [sec:future_directions]), and addressing this study’s limitations (in Section [sec:limitations]).

6.1 Future directions↩︎

Future work may extend the applicability and generality of our proposed framework. A natural next step is the release of CurricuLLM as a tool capable of extracting course listings, curriculum structures, and course descriptions from the websites of established cybersecurity programs. Such functionality would enable large-scale data acquisition and support comparative analysis across institutions.

Another direction involves enhancing the framework’s capacity to model a broader range of constraints on cybersecurity curricula, such as practical scheduling conflicts, and credit dependencies. Furthermore, CurricuLLM could be expanded to assess a student’s prior knowledge based on the completed parts of their curriculum, such that only courses of the appropriate level and with content not previously covered in their program are recommended.

Because CurricuLLM is not inherently constrained to the NICE workforce framework, it can be readily aligned with alternative workforce taxonomies, such as the European Cybersecurity Skills Framework [55]. More broadly, the pipeline could be adapted to other academic domains for which well-established curricular guidelines are available. For instance, the 2023 Computer Science Curricula [56] provides a suitable foundation for such an extension.

Finally, the weighting scheme used to represent the relative importance of Knowledge Areas based on regional demands (as introduced in Section 5.3.1) could be recomputed based on other geographical locations, such as those documented for the Finnish cybersecurity sector [57]. Furthermore, projected job market demands can be taken into account to support the development of curricula that remain aligned with evolving professional requirements.

6.2 Limitations↩︎

Although the CurricuLLM framework provides a structured approach for connecting curriculum content to workforce requirements, several limitations should be acknowledged. A central challenge in mapping curriculum content to Knowledge Areas is the inherently low inter-rater agreement observed among human annotators. Because the KAs partially overlap and are not mutually exclusive by design, human judgments often diverge, making it difficult to establish a definitive ground truth. This complicates both model training and evaluation. Consequently, the objective of CurricuLLM was not to maximize conventional metrics, but to approximate the level of agreement typically achieved by human annotators.

Another limitation arises from the fact that the current pipeline is exclusively based on course descriptions. While these descriptions are widely accessible and offer a practical entry point for large-scale data collection, they provide only a partial representation of course content. Other sources of information, such as learning objectives or sample educational material, could yield a more comprehensive understanding of the scope of the course and its content. However, such data is often unavailable due to copyright restrictions or institutional access constraints. Moreover, our reliance on course descriptions may introduce bias, as these descriptions are often designed to appeal to prospective students, and may overemphasize the societal relevance of the course rather than the core technical competencies covered.

7 Conclusions↩︎

In this work, we introduced CurricuLLM, a novel LLM-based framework to automate the analysis and personalization of cybersecurity curricula. Our approach directly addresses the gap between academic programs and industry needs by enabling data-driven alignment between curricula and industry standards. The core of our method is a two-tier approach: PreprocessLM transforms input data into a format suitable for machine learning and ClassifyLM classifies the data into the nine Knowledge Areas, where eight were defined by the CSEC2017 framework and one defined by [12]. Among several NLP methods evaluated, the BERT-based model demonstrated the best performance, providing a solid foundation for large-scale automated curriculum analysis. Our quantitative results show that CurricuLLM serves as a practical and effective tool for cybersecurity curriculum assessment. It accurately maps educational content to the CSEC2017 Knowledge Areas, achieving an inter-rater agreement comparable to human annotators, with minimal manual intervention or performance loss. Furthermore, we show how CurricuLLM can be applied at different levels of granularity, from individual courses to entire programs, to automatically identify the distribution of KAs within a curriculum.

An important aspect of CurricuLLM is its ability to evolve with the field, resulting in adaptive and workforce-aligned curricula. By integrating the newly released 2025 NICE Workforce Framework, CurricuLLM reflects the most recent definitions of cybersecurity competencies and work categories. This enables quantifying the job market relevance of individual Knowledge Areas based on current listings. By identifying which KAs should be emphasized in education according to current job market demands, we contribute to closing the cybersecurity workforce skills gap.

The methodological foundation developed for CurricuLLM, including dataset preparation, model training, and validation on real-world curricula, offers a systematic approach to applying LLMs to curricula development. Potential future work includes incorporating regional labor market data, expanding towards other related disciplines, and anticipating emerging skill demands. By bridging the gap between curricula and the dynamic cybersecurity workforce landscape, this work contributes to building more automated, industry-relevant, and personalized educational programs.

Acknowledgments↩︎

Funding for this work was made available by Cyber Security Education Cooperation Network Project Finnish Ministry of Education and Culture (Grant Number: OKM/60/522/2022), Swedish Research Council (VR) under grant 2021-05621, and the Swedish Knowledge Foundation (KKS).

All computations were executed on a Tesla T4 GPU, provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725.

Data Availability statement↩︎

All datasets used in this work, as well as the Python scripts used to generate our synthetic CSEC2017 dataset, are publicly accessible on our GitHub repository: https://anonymous.4open.science/r/LLMedu-227A.

Declaration of Interest statement↩︎

Arthur Nijdam reports financial support and equipment were provided by Swedish Research Council. Harri Kähkönen and Valtteri Niemi report financial support was provided by Finnish Ministry of Education and Culture. Sara Ramezanian reports financial support was provided by Swedish Knowledge Foundation (KKS). Paul Stankovski Wagner declares not to have any known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

8 Fine-tuning Strategy↩︎

In this section, our goal is to identify a suitable base model to serve as ClassifyLM and fine-tune it for the task of KA classification using our curated dataset. We detail the baseline methods analyzed (Section [sec:baselines]), the evaluation metrics used for comparison (Section [sec:metrics]), and the results of our experimental analysis of the candidate models (Section [sec:model_selection]).

8.1 Baselines↩︎

We consider the following NLP methods for the ClassifyLM task:

• Zero-shot Learning: In zero-shot learning, we simply prompt a LLM without domain-specific fine-tuning. We selected DeepSeek-V3 [51] and ChatGPT-4o-mini [23] for this baseline, since both models are widely available and relatively cost-effective. Additionally, we evaluate Qwen-2.5b-7B-Instruct [58] (the model adopted for PreprocessLM), to motivate that different models were used for PreprocessLM and ClassifyLM. The specific prompt used for zero-shot classification is detailed in Figure 10 in 11.

• Random Forest (RF): RFs are a well-known traditional machine learning method. In contrast to deep learning methods that learn directly from raw text, RFs require feature extraction. Here, we use both Bag-of-Words (BoW) and Term-Frequency-Inverse Document Frequency (TF-IDF), that assign numerical values to words based on how frequently they appear in the training set.

• LSTM: To establish a direct comparative baseline with the most relevant related work by Dzurenda et al. [42], we utilized the core architecture described in their paper. This model employs the standard BERT [21] tokenizer for text processing, followed by a unidirectional Long Short-Term Memory (LSTM) network for the final classification. Note that it was not possible to use Dzurenda et al.’s trained model directly, as it was optimized for classifying 12 European Union Agency for Cybersecurity (ENISA) work profiles rather than the 9 CSEC2017 Knowledge Areas. Therefore, we adapted their architecture and trained it on our dataset with the recommended learning rate of 0.001 for 20 epochs.

• Fine-tuned Transformer Models: We experimented with several transformer variants, namely BERT [21], ALBERT [59], RoBERTa [60], and DistilBERT [61]. BERT serves as a foundational benchmark. DistilBERT offers a more parameter-efficient alternative, reducing the model size by approximately 40% through knowledge distillation. RoBERTa improves upon BERT with optimized training techniques like dynamic masking and extended training, while ALBERT introduces mechanisms to minimize the memory footprint. All models were fine-tuned for a maximum of 20 epochs using a sigmoid output activation and the Binary Cross Entropy loss to accommodate the multi-label classification setting.

8.2 Metrics↩︎

The task of assigning course descriptions to one or more KAs is a multi-label, multi-class classification problem. While the accuracy provides an intuitive measure of overall prediction correctness, it is not well-suited for multi-label classification. It is highly affected by class imbalance: models can achieve a high accuracy solely by performing well on the KA with the highest number of samples. Accuracy also treats all errors equally and does not distinguish between false positives and false negatives, even though these could have different implications for curriculum analysis. Therefore, to enable a more nuanced performance evaluation, we do not use the accuracy, but instead compute the macro-averaged precision, recall, and F1 score. Here, ‘macro-averaged’ indicates that metrics are computed per KA and then averaged, such that KAs with fewer samples contribute as much to the final score as KAs with more samples. The definitions of the metrics used are as follows:

• Precision: The precision or true positive rate measures the proportion of correctly predicted positive labels among all predicted positives for a given class. We report the macro-averaged precision, which computes the metric independently for each KA and then takes the mean, assigning equal weight to all KAs. This is defined as: \[Precision = \frac{1}{C} \sum_{i=1}^{C} \frac{TP_i}{TP_i + FP_i}\] Here, \(TP_i\) represents the number of True Positives (TPs) for KA \(i\), while \(FP_i\) stands for the number of False Positives (FPs). \(C\) is the total number of classes, which is equal to 9 in the case of KAs. Macro-averaged precision yields a value between 0 and 1, where a high value indicates a low rate of false positives across the model’s predictions.

• Recall: The recall or sensitivity quantifies the proportion of actual positives that have been correctly identified as positive by the model. Analogous to precision, we report the macro-averaged recall: \[Recall = \frac{1}{C} \sum_{i=1}^{C} \frac{TP_i}{TP_i + FN_i}\] Here, \(FN_i\) stands for the number of False Negatives (FNs) for a given KA \(i\). The macro-averaged recall yields a value between 0 and 1, where a high recall can be achieved when the model has few false negatives.

• F1 score: The F1 score is the harmonic mean of the precision and the recall, addressing scenarios where one of these two metrics dominates. We report the macro-averaged F1 score, calculated as the mean of the per-class F1 scores: \[\text{F1} = \frac{1}{C} \sum_{i=1}^{C} \frac{2 \, TP_i}{2 \, TP_i + FP_i + FN_i}.\]

  The F1 score ranges from 0 to 1, where a higher value indicates superior performance by jointly considering precision and recall. Although the F1 score is the harmonic mean of precision and recall, its macro-average is not. Consequently, the macro-averaged F1 score can be lower than the macro-averaged precision and recall.

8.3 Model Selection↩︎

Table [tab:Baseline_results] gives an overview of the cross-validation scores of all baseline NLP methods mentioned in Section [sec:baselines], as computed on the fine-tuning dataset (described in Section [sec:datasets]). To ensure a robust performance estimate, we use k-fold cross validation. This technique partitions the fine-tuning dataset, consisting of synthetic CSEC2017 data and NIST Knowledge Descriptions as described in the previous subsection, into \(k\) non-overlapping folds. In each of the \(k\) iterations, the model is trained on \(k-1\) folds (corresponding to 90% of the data for \(k=10\)) and validated on the hold-out fold (the remaining 10%). In this way, it is possible to compute the performance of the model on the entire fine-tuning dataset. To balance the bias and variance trade-off associated with this process, the recommended setting for \(k\) is 10. We adopt \(k=10\) and split the training dataset into 10 non-overlapping 90/10 train/val splits.

We first evaluate zero-shot classification performance using Qwen-2.5b-7B-Instruct (which also serves as the backbone for PreprocessLM), and the state-of-the-art DeepSeek-V3 and ChatGPT-4o-mini models. An identical prompt, as indicated in Figure 10 in 11, was used for all models to ensure a fair comparison. As mentioned in earlier sections, we hypothesized that the tasks designed for PreprocessLM and ClassifyLM cannot be executed by the same model. From the data in Table [tab:Baseline_results] we can see that the Qwen-2.5b-7B-Instruct model does not have a good off-the-shelf performance for KA classification, a limitation that cannot be remedied via fine-tuning due to the small size of our dataset. The more powerful zero-shot DeepSeek-V3 and ChatGPT-4o-mini models also achieve low scores, demonstrating the limitations of general-purpose LLMs without domain-specific fine-tuning.

Random Forest (RF) based methods showed moderate improvements over zero-shot LLM prompting, which suggests that the features extracted from the fine-tuning dataset possess predictive value. Nonetheless, the limited performance of the RF models indicates that these models struggle to capture the complex semantics inherent to cybersecurity concepts. The baseline method proposed by [42], based on a unidirectional LSTM trained on pre-tokenized text, performed significantly worse than all other approaches. However, this can in part be attributed to the fact that the hyperparameter settings and training methodology were not optimized for our dataset, but set to the default values recommended by [42]. A bidirectional LSTM or a deeper neural network architecture could have improved the performance of the LSTM-based approach, but these architectural changes are beyond the proposed methodology of Dzurenda et al. and therefore considered outside of the scope of this paper.

In contrast, Transformer-based architectures substantially outperformed other approaches, with BERT achieving an F1 score of 0.64. We attribute this superior performance to the self-attention mechanism in transformer models, which allows them to weigh the importance of all words in a context simultaneously, capturing long-range contextual information more effectively than LSTMs or RFs. While transformer-based models generally outperformed traditional methods, BERT emerged as the best candidate. Based on these results, we therefore designate the fine-tuned BERT model as ClassifyLM.

9 Constrained Optimization for curriculum composition↩︎

After fine-tuning the ClassifyLM model as detailed in Section [sec:LL2_finetuning], it can be used to estimate the KA distribution of any given course based on its course description. The model produces a probability distribution over the CSEC2017 Knowledge Areas, providing a representation of the course’s cybersecurity coverage.

As established in Section [sec:role-basedweight], target requirements from the NICE framework for job roles, job categories, and the overall market, are similarly encoded as a probability vector over KAs. This shared vector representation enables the direct alignment of courses with cybersecurity workforce demands, framing the task of optimal course selection as a constrained optimization problem.

To design a curriculum aligned with job market requirements, it is important to consider the structural constraints of the curriculum. In particular, factors such as the total number of credits in the program, and the desired distribution between core and elective credits effectively constrain the optimization process. For example, the two-year MSc in Cybersecurity at KTH consists of 120 European Credit Transfer and Accumulation System (ECTS) credits, of which 39.5 ECTS correspond to mandatory core courses, 30 ECTS to conditionally elective courses, 30 ECTS are dedicated to the master thesis project and the remaining 20.5 ECTS are free choice electives. As presented in the case example (Section [sec:example]), CurricuLLM can be used to identify the optimal set of conditional electives for a specific target job role. The program offers 12 elective courses to first-year students, of which students are required to select at least 4. Although CurricuLLM can also be used to recommend free-choice electives, the present scenario focuses on optimizing the selection of conditional electives for simplicity. We formulate the optimization problem for the present scenario as a second-order constrained optimization problem, with the objective of aligning the student’s composed curriculum as closely as possible with the desired KA distribution of the target job role.

Here, we let

• \(m\): Number of KAs, \(k\): Number of elective courses to be selected, \(n\): Total number of available electives, \(c_M\): Total number of mandatory credits.

• \(c_i\) the credit attributed to course \(i\).

• \(E_{i,j} \in [0,1]\): Contribution of KA \(j\) to elective course \(i\). This is a normalized probability vector, such that \(\sum_{j=1}^mE_{i,j}=1\).

• \(T_j \in [0,1]\): Target KA distribution of the desired job role. This is a normalized probability vector, subject to \(\sum_{j=1}^mT_{j}=1\).

• \(x_i \in \{0,1\}\): Binary decision variable indicating whether elective \(i\) is selected.

• \(\mathcal{M} \subset \mathbb{R}^m\): The aggregated KA vector of the mandatory courses.

Such that the MIP formulation becomes \[\begin{align} \text{minimize} \quad & \sum_{j=1}^{m} \left| \frac{c_M}{c_M+\sum_{i=1}^nc_ix_i}\mathcal{M}_j + \frac{1}{c_M+\sum_{i=1}^nc_ix_i}\sum_{i=1}^{n} c_ix_i E_{i,j} - T_j \right| \\ \text{subject to} \quad & \sum_{i=1}^{n} x_i = k \\ & x_i \in \{0, 1\} \quad \forall i \in \{1, \dots, n\} \end{align}\]

Each course (mandatory or elective) is represented as a continuous \(m-\)dimensional probability vector indicating the proportional contribution of each KA to the course. Specifically, each elective \(i \in {1, \dots, n}\) is associated with a normalized contribution vector \(E_i = [E_{i,1}, \dots, E_{i,m}]^\top\). The mandatory curriculum component is similarly represented by a vector \(\mathcal{M} \in [0,1]^m\) and each elective course \(i \in \{1, ..., n\}\) has a KA contribution vector \(E_i\subset \mathbb{R}^m\). The desired job profile is represented as a target KA distribution \(T \subset \mathbb{R}^m\), obtained from Table 3 in [12]. Binary variables \(x_i\in \{0,1\}\) are introduced to indicate whether a given elective course \(i\) has been selected. The total number of electives selected is constrained to \(k\) using constraints (5) and (6). Additionally, because mandatory and elective courses carry different credit values at the program at KTH, we aggregate KA distributions in a credit-weighted manner. We let \(c_M\) denote the total number of mandatory credits and \(c_i\) the credits of elective \(i\).

The KA distribution of the student’s entire curriculum is then computed as the sum of the mandatory component, \(\mathcal{M}\), and the contributions of the selected electives: \(\sum_{j=1}^{m}\sum_{i=1}^{n}c_ix_iE_i\), weighted by the number of credits contained in each. The objective (4) is then to minimize the absolute deviation between the KA profile of the student’s curriculum and the target profile \(T\). This ensures that the selected electives fill the knowledge gap left by the mandatory curriculum as closely as possible.

We next examine to what extent this optimization problem should be reformulated to accomodate alternative curriculum constraints. The one-year MSc in Cybersecurity at Nanyang Technological University (NTU) consists of 30 Academic Units (AU), of which 12 AUs are reserved for mandatory courses and 6 AUs are dedicated towards a capstone project. The remaining 12 AUs are conditionally elective, where students choose four elective courses from a pool of nine. Because of the great structural similarity of this program with the program offered at KTH, CurricuLLM can be applied to the NTU master in a similar fashion as mentioned before.

The two-year M.S. in Information Technology - Information Security (MSIT-IS) offered at Carnegie Mellon University (CMU) consists of 75 core units, 36 free program electives, and 12 restricted electives. While the restricted electives can be chosen in a similar way as posed in the optimization formulation for KTH, the core units have a more complicated structure that we consider outside the scope of our current work. The core units are separated into a networking core (12 units, one course selected out of four options), an MSIT-IS core (12 units, one course selected from three options), a Security core (12 units, fixed), a Business and management component (12 units, comprising either two fixed courses or one alternative course), and a practical project worth 24 units. Additionally, we have not analyzed constraints such as the sequentiality of courses (e.g., Advanced calculus requires prior completion of Basic Calculus), practical constraints such as clashing timeslots, and the student’s prior knowledge.

10 Curriculum Analysis extended results↩︎

For each of the 3 curricula analyzed in Section [sec:job_to_KA] (KTH, NTU, CMU), we randomly selected 3 courses to be analyzed in further detail. Here, the results for the Building Networked Systems Security (KTH) course are provided earlier in the paper in Table [tab:networked_systems_security] in Section [sec:KA_to_job]. The rest of the results can be found in the following tables: Ethical Hacking (KTH), Table [tab:ethical_hacking], Language-Based Security (KTH), Table [tab:language_based_security], Network Security (NTU), Table [tab:network_security], Software Security (NTU), Table [tab:software_security], Privacy Preserving Technologies & Security in AI (NTU), Table [tab:privacy_ai_security], Advanced Real-World Networks (CMU), Table [tab:advanced_networks], Networked Systems Course (CMU), Table [tab:networked_systems_security], Information Security Policy and Management (CMU), Table [tab:security_policy_management].

For each course analyzed, we derived topics from the course descriptions using PreprocessLM, labeled them using ClassifyLM and present the final result of this process under the column CurricuLLM in the table. Then, Control Group A, consisting of 34 cybersecurity experts, and a group of three cybersecurity experts previously familiar with the CSEC2017 framework (noted as ‘Expert Group X’) were consulted to independently label the topics. The numbers listed in the tables down below represent these annotations, where the corresponding KAs for each number can be found in Table 2.

11 Zero-shot classification prompt↩︎

We employed zero-shot prompting of ChatGPT-mini-4o, and DeepSeek-V3 as a baseline for comparison with the proposed ClassifyLM structure (see Section [sec:baselines]). Zero-shot prompting requires the selected model to perform Knowledge Area annotation based solely on the instructions and definitions provided in the prompt, without any prior examples. The prompt provided to both ChatGPT-mini-4o and DeepSeek-V3 is detailed in Figure 10.

None

12 KA distribution NICE Job Roles↩︎

The relative importance of each of the nine Knowledge Areas for all work role categories defined in the NICE framework v2.0.0 is shown in Figure [fig:job_role_KAs_combined] and discussed in Section [sec:job_to_KA]. The ‘relevance’ of KAs was derived by collecting the KDs associated with a given job role, annotating them with corresponding KAs using ClassifyLM, and computing the resulting distribution of KAs. For completeness, this appendix provides the relative importance of the KAs for all individual NICE work roles. These detailed results are displayed in Table [tab:nice_work_roles].

13 Knowledge Area 0: Miscellaneous↩︎

In [12] the authors identified seven knowledge units in the knowledge area 0: Miscellaneous. The description of these knowledge units are as follows.

• Computer Science: includes topics related to Programming and Software Development, System Architecture, Database Management, Mathematics, Information Systems, and Human and Organizational Aspects of Information Technologies.

• Business and Law: includes topics related to Regulatory Frameworks, Organizational Operations & Policies, Stakeholder Relationships, and Management & Human Resources.

• Communication and Networking: includes topics related to Telecommunications Fundamentals, Network Architectures Fundamentals, Network Management, Communication Systems, and Information Gathering and Dissemination.

• Information Technology: includes topics related to Technology Infrastructure, Content Systems, Technology Trends & Integration, and Media, Communication, & Dissemination.

• Cyberspace Practice: includes topics related to Adversarial Models, Cyber Attacks & Vulnerabilities, Cyber Operations & Practices, Core Cyber Concepts & Principles, and Security Tools, Vendors, & Defensive Technologies.

• Pedagogy: includes topics related to Learning Frameworks, Educational Technology, Training Policies, Organizational Education Practices, Assessment Techniques, Experiential & Applied Learning, and Psychology, Behavior, & Human Factors.

• Intelligence: includes topics related to Intelligence Fundamentals, Intelligence Frameworks, Intelligence Collection, Intelligence Management, Targeting & Operational Intelligence, Intelligence Production, Intelligence Dissemination, Intelligence Planning, Resource Prioritization, Partner & Organizational Intelligence Integration, and Covert & Specialized Intelligence Techniques.

References↩︎

---

1. It is worth noting [16] also proposed a system named CurricuLLM. Their work revolves around curriculum learning, a technique used within Reinforcement Learning where agents are trained on progressively harder environments, and is unrelated to our approach.↩︎