¹

1 Introduction↩︎

System administrators and operators find themselves required to survey, design and secure environments constructed from an ever-expanding ecosystem of architectures, specifications, frameworks and tools. The public and hybrid cloud paradigm has handed a varying degree of these responsibilities over to hyperscalers, depending on the number and type of businesses’ services an IT company may host there. However, the adoption of the private cloud approach has grown over the last few years [1]. Whilst using hybrid, multi-cloud environments is the norm, the investment in private cloud infrastructure is increasing at twice the rate of public clouds [2]. Some of the adduced reasons for this shift include increasingly stringent regulatory compliance, translating into data sovereignty and isolated workloads (e.g. AI) on premises.

These hybrid and private cloud approaches require organisations to retain in-house knowledge regarding general and security-related configurations. Modern tooling used in both local and hybrid setups becomes more complex as it encompasses more moving parts that must be continuously managed (e.g. tools for granular and stricter identity management, authentication and authorisation) [3]. At the same time, the scarcity of qualified cybersecurity experts [4] makes it more difficult to properly assess the significance and impact of adopting security-related tooling, not to mention the interpretation of the security control frameworks and threat modelling. While methodologies like DevSecOps aim to involve all actors to reduce the threat surface early in the software development lifecycle, setting up and hardening operational environments with third-party tools is more fragmented and prone to error.

Because of that, teams without proper security expertise face a three-fold problem: (i) deciding which security controls to survey and prioritise first from a large, abstract pool; (ii) selecting subsets of controls that best mitigate their organisation’s requirements and processes; and (iii) optimising this selection to minimise both the vulnerable attack surface and the invested time, effort and budget. In some occasions, this task is complex even for cybersecurity experts, as the optimal control selection requires reaching a trade-off between under- and over-provisioning from vast catalogues [5] with varying contributions, which makes a comprehensive survey even more complex. Specifically, the vulnerability surface of the organisation must be maintained or reduced, whilst ensuring minimal resources are committed to it. A proper balance ultimately minimises costs, and is thus also reviewed from the economic side [6].

The above complexity can be tamed by introducing Security Decision Support Systems (DSS). Such systems guide operators during the initial surveying stage, bridging the knowledge gap by internally evaluating the security control frameworks according to formalised requirements. By providing tailored, optimised recommendations, a Security DSS allows operators to harden their environment more efficiently by maximising compliance and minimising invested time and effort.

1.1 Contributions↩︎

This work proposes a Security DSS that embodies expert knowledge as a Knowledge Graph (KG) and a Game Theory framework that recommends minimal security control sub-families to best meet the requirements over the security dimensions. These dimensions represent the system’s data protection objectives, encompassing the CIA triad (confidentiality, integrity, availability) and others (authenticity, accountability, non-repudiation, privacy) [7], [8]. The game evaluates the agents’ utility functions while exploring the decision space to maximise required security coverage and minimise under- and over-provisioning. Finally, a minimal set of best-suited control sub-families is identified and recommended. The following make this possible:

1. A curated and unified dataset that maps the Security and Privacy Control Families (SPCFs) and Sub-Families (SPCSFs) from NIST SP 800-53 rev5 [9] to 38 columns with data of interest: their contribution to cover the security dimensions, to mitigate threat vectors from the STRIDE-LM model; as well as detailed and aggregated scores from other works measuring effectiveness on mitigating attacker’s techniques and scores from expert surveys, among others described in Section 3.1.

2. A Multi-Agent Influence Diagram (MAID), as an extension to classical Bayesian Network (BN) [10] models. These encode the coverage per security dimension and the total score from the dataset above as a KG with embedded probabilistic influences across random variables within MAID nodes.

3. A multi-agent, non-zero-sum, simultaneous game-theoretic framework using no-regret online learning algorithms to heavily reduce the computational complexity of exploring the decision space. It evaluates the agents’ (security dimensions) utilities from the MAID’s KG and maximises the overall payoff, returning the joint profile that best meets the requirements. This is used as upper boundary by the recommender stage to maximise accuracy while minimising over-provisioning.

The remainder of this paper introduces first the related work in Section 2. The main sections detail the contents and generation process of the curated security dataset in Section 3 and the design and implementation of the MAID-based security recommender that leverages such dataset in Section 4. Section 5 compiles the evaluation methodology, metrics and tests of the security recommender and its baseline. Section 6 concludes with an overview of the system along with its assumptions and identified limitations for both the security dataset and recommender; as well as the concluding remarks and future work directions.

2 Related work↩︎

Related academic literature is documented for each contribution from this work: (i) assessing and modelling the impact of the security controls available to users; and (ii) modelling the security decision support problem to find suitable recommendations.

2.1 Security control’s importance evaluation↩︎

For a security decision process, a knowledge base must exist that (i) categorises the different security mechanisms or controls to enact; and that (ii) scores these appropriately to the task at hand, i.e. depending on specific criteria. Three academic works were identified that perform an evaluation of security mechanisms, whether using its own taxonomy or basing on existing ones, such as the list of SPCFs and SPCSFs provided by NIST SP 800-53 rev5 standard [9]. This list contains (i) 20 families (SPCFs); (ii) 300 active intermediate controls, or sub-families (SPCSFs); and (iii) 714 active control enhancements. Whilst other standardised security and privacy controls catalogues exist, such as ISO/IEC 27002:2022 [11] or MITRE D3FEND [12], NIST SP 800-53 is chosen for its higher relevance in previous literature, its periodic revisions and its open availability both from reports [9] and repositories [13] (whose common format allows for easy programmatic analysis).
To identify the most impacting security controls, [14] performed a three-round interview with cybersecurity management layer experts. This work provides its own taxonomy for the security controls and provides scores and a mapping towards NIST CSF 2.0, used to improve organisation’s cybersecurity plans. In [15], a simplified taxonomy of functional security dimensions was produced with five categories of security controls. The authors analysed which controls could be implemented by software and provided code examples for popular security frameworks. The NIST SP 800-53 SPCSFs had its effectiveness assessed in [5], based on their ability to mitigate 188 adversarial techniques from the MITRE ATT&CK framework. This study considered 7 metrics per SPCSF, where Technique Coverage (TEC) is the prominent one. They filtered the SPCSFs by relevance (107 with \(TEC > 0\)), they applied k-means clustering to separate the top 21 critical SPCSFs that perform the best (i.e. more risk-efficient and versatile) for organisations to prioritise.

2.2 Probabilistic security and risk decision support↩︎

Both probabilistic and game-theoretic approaches are used in recommender and decision support scenarios; starting from the more probability-oriented classification and suggestion and incorporating also the motivation of different agents that are to converge. The latter typically consider Nash Equilibrium [16] as a stable state reached when agents cannot improve their outcome by acting unilaterally — yet others exist, such as Stackelberg, subgame perfect, epsilon or correlated equilibria.

Probabilistic models, such as those based on the Bayes’ Theorem, allow updating the probability of a hypothesis, given its likelihood and the conditioned probability of some evidence (known data) given that hypothesis. Bayes’ theorem is at the core of BNs [10], [17], which allow modelling and updating events (as random variables, depicted as nodes) based on known information or beliefs (evidence) and use these to infer the events’ unknown joint probabilities (hypothesis) conditioned on selected subsets of evidence. BNs can leverage this for recommender systems. In [18], an adaptive DSS for cybersecurity incident mitigation leverages the Directed Acyclic Graph (DAG) within a BN as well as a custom, domain-specific model data for better computational efficiency. It applies Bayesian confidence calibration on well-known vulnerability and exploit scoring systems to compute the exposure probability, applying multi-objective optimisation for three pillars (attack likelihood, impact severity and system availability) to recommend mitigating controls. [19] proposes a cybersecurity DSS that converts the online optimisation to Mixed-Integer Conic Programming and runs over probabilistic attack graphs using Bayesian Stackelberg Games. They model the defender (leader) and attacker (follower) and account for the uncertainty on the latter’s capabilities; ultimately selecting a subset of security controls from 24 defined from their university, so to counteract potential or ongoing attacks.

Game Theory approaches can also be used to model systems that consider the behaviour of agents under either collaborative or competing incentives and propose the most strategically beneficial behaviour. Some authors combine these with Influence Diagrams to provide more descriptive and versatile frameworks, and at the same time, ensure fairness across involved agents and overall stability. A MAID is used in [20] to model an attacker-defender environment, modelling four categories of custom counter-hybrid measures that range from the technical to the policy level; then calculating the subgame perfect equilibrium that balances each agent’s payoff and ultimately analysing results per counter-hybrid measure. In [21], a game-theoretic DSS evaluates the effectiveness of strategies used by an analyst to protect according to the attacker’s objective. It uses a 2-player (security analyst and attacker), zero-sum, one-shot algorithm, considering attacker profiles and filtering on budget effectiveness. The end-user is left to provide the security controls’ catalogue and expert data evaluation. Another game-theoretic approach to cybersecurity decision is provided in [22], where a 2-player game models two competing countries and guides them on the optimal strategy regarding disclosure of vulnerabilities. In [23], attacker and defender populations are modelled as a 2-player asymmetric, non-cooperative, non-zero sum game where each can either act or not. Based on Evolutionary Game Theory, this helps investigating the dynamics of the players’ interactions, including the costs, benefits and defence probability and providing suggestions for the organisations’ defence. Adversarial Risk Analysis (ARA) is a related approach that allows decision making with strategic opponents and uncertain outcome; modelling different types of (non collaborative) agents. Authors from [24] propose a risk analysis framework for AI-based systems, modelling both the defender’s and attacker’s expected utilities with multiple attributes. They consider, among others, the portfolio costs and organisational risk attitude for the defender; and follow a Bayesian probabilistic approach with Monte Carlo sampling on the attacker to model its uncertainty. With this, potential portfolios of (generic) cost-effective security controls are suggested. On the ground of forensic investigations against strategic adversaries, [25] proposes a two-player (investigator and attacker), non-zero-sum game that finds the Nash Equilibrium through the Lemke–Howson algorithm [26]. Utility functions model benefits and costs per actor and attempt to maximise each actor’s payoff, considering the other actor’s actions: uncovering the attack (investigator) or hide traces (attacker). A multi-stage cyber attack is modelled as multiple Bayesian games of incomplete information that use a graph of attack actions. Bayesian probabilities also model here the uncertainty regarding the type of attacker, and the ultimate goal is to generate an optimal strategy for the investigator that also balances the attacker’s type and the potential benefit and cost per action.

2.3 Gap analysis↩︎

When assessing the security controls’ contribution for a security decision scenario, the current literature shows significant fragmentation between taxonomies [15] that capture relations and the SW-implementable assessment, expert managers’ surveys [14] focusing on cybersecurity management and maturity assessment and datasets providing several technical metrics per control [5] and mappings to threats, yet not exhaustively computed for all the security catalogue in use. This work combines relevant attributes (such as score and SW-implementability) from the previously indicated academic datasets into a unified one. It also integrates other attributes from standardised catalogues of security and privacy controls and also mapping controls to their impact on threats, following well-known threat modelling. This considerably enriches the resulting dataset, providing enough granularity (per SPCSF) to facilitate the construction of KGs.

Besides this, applying these full datasets as KG in game-theoretic models to obtain exact, optimal results (or equilibria) introduces considerable scalability issues. Since finding exact equilibria can be Polynomial Parity Arguments on Directed graphs (PPAD)-complete [27], such computational complexity restricts analysed implementations to 2-player games with small decision domains [19]–[25]. Without simplifications like zero-sum games [21], obtaining exact optimal solutions using Nash, Stackelberg or subgame perfect equilibria becomes directly intractable as the decision space grows, as influenced from e.g. number of agents or decision domains [19]–[21], [25]. To bypass exact calculations, other approaches rely on iterative Monte Carlo simulations [24], which sample random utilities to estimate the attacker’s actions but can incur heavy computational overhead for large control sets; as well as on Evolutionary Game Theory [23], which handles probabilities of interactions at population-level and thus can lose significant granularity.

Overall, the literature does not adequately cover the combination addressed in this work. Existing works either evaluate security controls from specific perspectives (e.g. expert judgement, implementability or threat mitigation) or model cybersecurity decisions through probabilistic and game-theoretic approaches that typically focus on smaller control sets, two-player (e.g. attacker/defender) setups, or cost-driven objectives. What remains insufficiently addressed is a knowledge-based framework that integrates heterogeneous control-level evidence into a unified representation, expresses requirements in terms of security dimensions and delivers granular objectives while supporting scalable multi-agent recommendation of Security and Privacy Control Sub-Families. Therefore, this work models a non-zero-sum, simultaneous game with granular interactions across 7 agents. By relating each security dimension to an agent, the game can account for the differentiated interactions (e.g. cooperative or competitive) across each type of agent. By avoiding exact calculations and applying instead no-regret online learning algorithms, this approach bypasses the PPAD-complete scalability issue and offers a scalable approach that converges faster on approximate optimal joint profiles from the explored decision space and refines these into optimal solutions. Consequently, the resulting DSS runs in up to 35.7 seconds. The KG is recreated from the latest available dataset for every execution; thus maximising operations on an up-to-date dataset.

3 Security dataset↩︎

A security dataset is first constructed to provide relevant information for the security controls. This dataset acts as the cornerstone of the proposed Security DSS, as its information feeds the different decisions and information modelling present in the DSS architectural building blocks. Fig. 1 illustrates the relations (dependencies) across these blocks, their mapping to the three described algorithms and their order of execution as part of the Security DSS.

The structure and contents of the dataset are explained below, along with its generation process and the analysis of the resulting data.

3.1 Structure↩︎

This dataset relates security controls (SPCSFs) from well-established frameworks in the cybersecurity community to both existing expert- and ML-generated scores provided by academic works. It also performs a simple quantification process on the SPCSF’s impact on mitigating threats and on covering or addressing the seven security dimensions, as taken from well-known open Information Security (InfoSec) sources. With these scores and information, three partial and one global scores are generated per SPCSF.

To extract security controls highly relevant to the community, the NIST SP 800-53 rev5 [9] catalogue was chosen. Beyond its \(\sim\)​20-year life and its adoption in both engineering and academic communities, it is accessible in machine-friendly formats that facilitate automated processing, e.g. OSCAL [13]. It is structured in 20 control families, with each family containing multiple controls, here called SPCSFs; and each control potentially having multiple control enhancements. These SPCSFs families comprise specific security topics, from access control (AC) to Supply Risk Management (SR). It contains 324 SPCSFs (300 of these not being withdrawn). These conform the rows of the proposed dataset \(DS_{8}\).

Table 2 groups the 38 columns available in \(DS_{8}\), indicating the number of features or columns per group, the originating source (\(DS_{i}\)) and arrows indicating which were used to derive aggregated partial and total scores. All sources used to construct the dataset \(DS_{8}\) are identified as \(DS_{i}, i \in [1,7]\) and summarised in Table 1. The data transformations used to generate the dataset are detailed in Algorithm 3. Overall, the generated dataset \(DS_{8}\) contains 300 SPCSFs and 38 columns with extracted and generated data properly correlated.

3.1.1 Source datasets↩︎

As indicated in Table 2, three types of datasets were used: from InfoSec communities, from academic papers and GenAI-interpreted.

Three academic works, introduced in Section 2.1, were used in this contribution to deliver averaged scores for SPCSFs. These are (i) expert-based scores per SPCSF in \(DS_{2*}\) [14]; (ii) taxonomy and SW-implementability criteria in \(DS_{6}\) [15]; and (iii) the analysed mitigation impact, considering the MITRE ATT&CK framework in \(DS_{7}\) [5]. InfoSec sources from NIST were also used; both from (i) the OSCAL catalogue (\(DS_{1}\)) [13] and from (ii) the CyberSecurity Framework (CSF) (\(DS_{3}\)); as well as from the CSFtools site [28] for both (iii) the impact of the NIST 800-53 rev5 SPCSFs on mitigating Microsoft’s STRIDE-LM threat vectors (\(DS_{5}\)). GenAI was used to analyse the coverage of each SPCSF towards the security dimensions (\(DS_{4}\)), based on the text-based description from the InfoSec source (i).

As introduced above, two well-known threat frameworks are used in \(DS_{8}\). One of them is the MITRE ATT&CK, considered from \(DS_{7}\) and involving a taxonomy of adversarial tactics composed of techniques, as well as detection strategies and mitigations per technique. The other one is Microsoft’s STRIDE-LM, providing seven threat vectors that can be leveraged by an attacker to compromise an environment: spoofing (S), tampering (T), repudiation (R), information disclosure (I), denial of service (D), elevation of privilege (E) and lateral movement (LM).

3.1.2 Dataset attributes↩︎

The columns provided in \(DS_{8}\) contain mostly attributes correlated from the other \(DS\) sources. The grouped columns from Table 2 are explained below.

The two NIST columns provide the identifier and name per SPCSF for most \(DS\) (except for \(DS_{2*}\)), which are used for direct (or indirect) correlation across datasets. Two other columns from \(DS_{7}\) provide the NIST baseline and priority from both NIST SP 800-53 rev5 and rev4, respectively. The priority was removed in r5, yet is deemed of interest here. The interpretation of the implementability through SW of a SPCSF is provided in three columns that are either inferred from attributes from \(DS_{1}\) or directly extracted from \(DS_{4}\) and \(DS_{6}\). The contribution (coverage) of each SPCSF to the seven considered security dimensions is provided in \(DS_{4}\) as real values between 0 and 1, with the constraint that all of them sum up to 1 for normalisation. This comes from interpreting the SPCSF descriptions from OSCAL \(DS_{1}\) and CSFtools site [28], which was left to a GenAI model. For human inspection, an extra column was provided with its justification of the interpretation. Another measure of impact is the contribution of each control enhancement per SPCSF to cover the seven threat vectors from the STRIDE-LM framework. These columns are included in \(DS_{5}\) and contain a list of comma-separated SPCSF’s control enhancements. A custom taxonomy column coming from \(DS_{6}\) is added to potentially help establishing categories of SPCSFs depending on their kind of actions and benefit. Six columns are used to denote the impact of SPCSFs as mitigating mechanisms, considering the MITRE ATT&CK framework. These are fetched from \(DS_{7}\) and encode varying metrics, such as the number of techniques or mean number of tactics covered. Finally, scores are derived from previous attributes. Three partial scores are calculated (i) from the scores given by cybersecurity experts(\(DS_{2*}\)), by averaging these; (ii) from STRIDE-LM (\(DS_{5}\)), the mitigation efficacy is computed using a simple ratio based on the number of controls enhancements involved in mitigation from all available in each SPCSF; and (iii) from MITRE ATT&CK (\(DS_{7}\)), the mitigation efficacy averages its metrics after performing on-the-fly normalisation. The total score is obtained by averaging these three partial scores.

The model detailed in Section 4 uses the NIST identifiers, the SW-implementability criteria, the security dimension coverage and the total score — obtained from the aggregated values in the partial scores described in Table 2.

3.2 Generation process↩︎

First, the NIST OSCAL catalogue [13] is processed to identify (i) active SPCSFs per SPCF group (\(g_{a}\)) and (ii) SW-implementable criteria per SPCSF, determined by OSCAL labels and depending on whether the SPCSF is not withdrawn from previous releases and it is labelled as either "implementation-level" or "system" – but not as "organization", which are deemed more strategic. These are covered in Algorithm 3, lines [alg:implementation-dataset:dataset:line:s1]-[alg:implementation-dataset:dataset:line:s5].

Now, the data for each SPCSF from the covered works is correlated and aggregated. First, the two datasets from \(DS_{2*}\) (\(DS_{2A}\), \(DS_{2B}\)) [14] are merged, using the common "process_id" column. These hold (i) the security ontology resulting from the interviews as one dataset; and (ii) the score for each category in the security ontology as another one. This is covered in Algorithm 3, line [alg:implementation-dataset:dataset:line:s6]. It is worth noting that the "merge" function is implemented as a left join, which takes as ground truth all items at the first (left) parameter.

Then, the \(DS_{4}\) mapping is merged into the final dataset, considering the full list of SPCSFs (Algorithm 3, line [alg:implementation-dataset:dataset:line:s7]). \(DS_{4}\) was manually compiled upon querying the Gemini 2.5 Pro model on the CSFtools [28] reference to identify the contribution (a real number between 0 and 1) of each SPCSF towards the fulfilment of the security dimensions: confidentiality (C), integrity (I), availability (A), authenticity (Au), accountability (Ac), non-repudiation (Nr) and privacy (Pr). While relying on expert knowledge per SPCSF yields a more accurate operational ground truth, this LLM-generated data serves well the primary objective of validating the appropriateness of the framework across varying values. In operational environments, cybersecurity experts should replace these values upon a manual, detailed approach; e.g. inspired by guidance from authoritative sources such as CNSSI 1253 [30] or from programmatic, statistic analysis of descriptions from the OSCAL catalogue [13].

After this, the final dataset is extended with a mapping (\(DS_{5}\)) that stores the contribution of each SPCSF (and internal control enhancements) towards the mitigation of each STRIDE-LM threat vectors. This mapping was obtained manually per SPCSF in CSFtools [28] and merged in Algorithm 3, line [alg:implementation-dataset:dataset:line:s8]. A second criterion to determine the SW-implementation, as provided by \(DS_{6}\) [15], is also integrated here (Algorithm 3, line [alg:implementation-dataset:dataset:line:s9]).

At this point, the previously generated \(DS_{2}\) is merged into the final dataset. To do so, the \(DS_{3}\) mapping is used for correlating keys; as it relates controls from NIST SP 800-53 r5 and those from NIST CyberSecurity Framework (CSF) 2.0 [29]. In particular, the ID of each SPCSF in the final dataset is compared against each item from the "reference_document" column in \(DS_{3}\). If these match, the SPCSF’s family (i.e. its SPCF) is retrieved. The available statistical metrics from \(DS_{2}\) (i.e. 36 processes from its custom taxonomy) are incorporated by using both (i) the internally provided mapping between this custom taxonomy to NIST CSF 2.0 (column "mapping_nist_csf", post-processed in the "filter_fam" method) and (ii) the \(DS_{3}\) mapping described above ("focal_document" column). These statistical metrics consider the mean, median, mode, Standard Deviation (SD) and Inter-Quartile Range (IQR). This is covered in Algorithm 3, lines [alg:implementation-dataset:dataset:line:s10]-[alg:implementation-dataset:dataset:line:s15].

\[\label{eq:implementation-dataset:generation:min-max-scaling} mms(\vec{S}) = \frac{s-s_{min}}{s_{max}-s_{min}} ~ \forall ~ s \in \vec{S}\tag{1}\]

The last block in the algorithm performs a series of calculations to (i) provide a single score from the considered academic works \(DS_{2}\) [14] and \(DS_{7}\) [5]; (ii) generate a score for the STRIDE-LM contribution; and (iii) average all 3 scores, conditioned to any of the SW-implementation criteria (\(impl_1\) or \(impl_2\)) being true. First, each statistical set of metrics from \(DS_2\) is first normalised applying the \(mms\) function for min-max scaling, as per Eq. 1 , to ensure that each value \(s\) inside meets \(0 \le s \le 1\). Then, the value of each statistical metric for any given SPCSF is taken and aggregated into the intermediate \(d_{8i}.score_{DS_2}\). This takes place in Algorithm 3, line [alg:implementation-dataset:dataset:line:s17].

1.5pt

4pt

Then, values from \(DS_{7}\) are incorporated. It is worth noting that, before this process can start, the publicly available dataset for \(DS_{7}\) [5] seemed to require minor modifications in the publicly provided scripts in order to generate the full dataset. To that end, both the "main" and "utils" files were modified to (i) properly compute the severity centrality from the given patterns; (ii) use the latest file names; and (iii) extract all values from the clustering method. With that covered, all but one effectiveness custom metrics (\(DS_{7}.cm\)) provided by \(DS_{7}\) [5] are used; which makes it 6 out of the original 7 metrics. Such metric is the Tactic Coverage (TAC), which provides granular scalars per SPCF that are already considered in the already used mean TAC (MTAC) metric. These values are handled similarly to the \(DS_{2}\) ones: first normalised using the \(mms\) function, then averaged (Algorithm 3, line [alg:implementation-dataset:dataset:line:s18]). Besides the custom metrics, this dataset incorporates the NIST-defined baseline and priority metrics (\(DS_{7}.nm\)) per SPCF; which are used to prioritise the implementation of a given control depending on the baseline to cover (i.e. lower must be implemented first) and priority, both of which range from 0 to 4. Each of these values per SPCSF is averaged and normalised simply considering its maximum value (Algorithm 3, line [alg:implementation-dataset:dataset:line:s19]).

Then, both custom (\(DS_{7}.cm\)) and NIST (\(DS_{7}.nm\)) metrics are averaged in Algorithm 3, line [alg:implementation-dataset:dataset:line:s20]. It is worth noting that both the CR (\(cm\)) and the NIST priority (\(nm\)) metrics follow essentially different approaches than other metrics, since smaller values denote higher importance there; and must thus be transformed to convey a similar approach whilst still keeping their significance.
The third and final score relates to the contribution towards the mitigation of the STRIDE-LM threat vectors, previously introduced from \(DS_{5}\). In Algorithm 3, line [alg:implementation-dataset:dataset:line:s21]; each SPCSF is iterated to count how many of its main (e.g. AC-9) or its control enhancements (e.g. AC-9(3)) contribute towards mitigating the STRIDE-LM’s threat vectors. That number is divided by the total number of control enhancements available per SPCSF, which is programmatically obtained from the OSCAL catalogue \(DS_{1}\) – and adding one more to count also the SPCSF itself. This provides a measure of the effectiveness of a given SPCSF into mitigating such threats.
Finally, the three scores above are averaged and then the final score per SPCSF is calculated, conditioned on whether either one or the other implementation criteria (\(p_{i}\), considering both \(impl_{1}\), \(impl_{2}\)) is met. The generated dataset, correlated from the previous source, is available at \(DS_{8}\).

3.3 Composition analysis↩︎

Table 2 enumerates the top 20 SPCSFs that are SW-implementable, ordered from highest to lowest and according to the total score calculated in this work (Algorithm 3, line [alg:implementation-dataset:dataset:line:s23]). It shows each SPCSF, its total score and the degree of contribution from each of the contemplated works and datasets. Additionally, a colour code is used to represent if any given SPCSF is also present in the top 20 ranked controls of each considered dataset by showing its colour: blue for #2 [5], yellow for #3 [14] and green for #4; as well as gray for contributions from 2 or more datasets. Those without colour indicate that these are still contributed from these other datasets, yet further down in their ranked list. There, the highest contributing SPCFs are SC, AC and SI with an average of 9.25, 4.25 and 2.5 contributed SPCSF items, respectively. Some SPCF categories (AT, CA, IR, MA, MP, PE, PL, PM, PS, PT, SA) are contributed only by non-SW-implementable SPCSF items that relate to awareness, assessment, authorisation, monitoring, incident response and management, among others. The rest of the categories (AC, AU, CM, CP, IA, RA, SC, SI, SR) are considered instead; contributing to access control, accountability, configuration management and protection efforts, among others. More information can be found in the dataset’s ancillary files, such as the top 20 total (i.e. SW-implementable or not) SPCSF –also in Fig. [fig:implementation-dataset:dataset-score-all]– and the number of contributing SPCSF controls per SPCF and their statistical metrics and spread. Fig. [fig:implementation-dataset:dataset-score-implementable] complements this by placing each SPCSF into each of the 10 deciles (in the Y-axis), depending on its total calculated score.

Besides the score distribution, Table 3 and Fig. 5 show the number and IDs of the SPCSFs contributing to the security dimensions, considering both the CIA triad with confidentiality (C), integrity (I) and availability (A); along with authenticity (Au), accountability (Ac), non-repudiation (Nr) and privacy (Pr). Fig. 5 renders the contribution of all 20 (Fig. [fig:implementation-dataset:dataset-secfeats-all]) and the 9 SW-implementable (Fig. [fig:implementation-dataset:dataset-secfeats-implementable]) SPCFs.

Finally, Table 4 summarises some relevant data. From the total 300 SPCSFs, 97 are considered SW-implementable. Regarding the contribution of the considered works towards these SW-implementable SPCSFs, #2 [5] is the most comprehensive, as its measurements yield \(> 0\) for 96 out of these 97; whereas #3 [14] provides 53 and #4 considers 47, as per the "Contribution" columns of Table 4. The minimum, average, mode, maximum, SD and IQR statistical metrics are provided right after for the SW-implementable SPCSF items belonging to each SPCF category. As observed in Fig. [fig:implementation-dataset:dataset-score-implementable], the SC, AC and SI SPCFs are the most prevalent in the highest deciles (roughly around 6 to 9) for the top 20 SPCSFs from Table 2.

3.3.1 Key findings↩︎

From the analysis of the dataset’s content, the following considerations are highlighted:

• The highest scored SW-implementable SPCSFs lie in the SI, SC, IA, AC and CM SPCFs (deciles 6 to 9); with the majority coming from SC. Unsurprisingly, these cover data (e.g. configuration) integrity, protect system communications and control access to data for confidentiality. This is consistent with \(DS_{7}\) [5], whose top 10 controls (per \(TEC\) metric) consist only of these SPCSFs and which covers also 85% of the whole dataset’s upper \(TEC\).

• Considering the SW-implementable SPCSFs, the best covered security dimensions are confidentiality and integrity, followed by availability and authenticity; whereas the least covered ones are privacy, accountability and non-repudiation. The SW-implementable SPCFs with higher overall coverage are SC, AC, IA and SI.

• When considering total SPCFs, some SPCFs cover well a specific security dimension that cannot be covered by contained SW-implementable SPCSFs (e.g. the total number of SPCSFs in PE and PM covers availability well), but the implementable SPCSFs that can best cover it belong to SC and SI instead — yet with a high scoring variability. This highlights the direct impact of non-SW-implementable SPCSFs (on planning, physical and personal data protection) to ensure data availability and privacy. On the other hand, these non-SW-implementable SPCFs usually score low in the aggregated \(DS_{7}\) [5] data and often also in the final score.

• The IA and SC SPCFs (identification and authentication & system and information integrity) contribute towards authenticity (with scores ranging between 0.8-0.5 and 0.5-0.1, respectively); no matter whether considering total or SW-implementable SPCSFs.

4 Security recommender↩︎

The knowledge-based multi-agent framework, explained below, leverages the previous dataset \(DS_{8}\) to encode as KG in the MAID model \(\mathcal{M}\) that recommends SPCSFs.

4.1 Background↩︎

MAIDs [31] extend from Influence Diagrams (IDs) [32], and IDs extend themselves from BNs [33]. On the one hand, BNs \(B = (G, \mathbb{P})\) are represented with (i) a Directed Acyclic Graph (DAG) \(G = (V, E)\) encoding random variables (events) \(X_{i}\) as vertices and the influence across these as directed edges; and (ii) Conditional Probability Distributions (CPD), encoding the probability \(\mathbb{P}\) of an event \(X_{i}\) occurring based on that of its parent nodes \(Pa(X_{i})\), i.e. \(\mathbb{P}(X_{i}|Pa(X_{i}))\). On the other hand, IDs consider (i) the nodes \(V\) in BNs, representing \(X_{i}\), as chance nodes \(C\), and also include two new types of nodes: (ii) decision variables, for the agents to select as part of their strategy; and (iii) utility variables, to represent the agent’s preferences. Thus, the resulting graph could be formalised as \(G = (C, D, U)\) and can be used to more easily analyse the probabilistic dependencies across \(X_{i}\) (\(C\)) and \(D\). A MAID \(\mathcal{M} = (G, \theta)\) [34], [35] consists of (i) a MAID \(G = (\mathcal{A}, V, E)\), where \(A\) is the set of agents and \((V, E)\) is a graph defined by nodes \(V = (C, D, U)\) to represent chance, decision and utility nodes, respectively, and edges \(E = V \times V\); along with (ii) a parameterisation \(\theta\) that contains, per node in \(V\), a finite domain \(\text{Dom}(X_{i}) \in \mathbb{R} ~\forall~ X_{i} \in V\); a real domain \(\text{Dom}(X_{i}) \in \mathbb{R} ~\forall~ X_{i} \in U\); and CPDs for chance \(C\) and utility \(U\) nodes. \(\text{Dom(}X_{i}\text{)}\) will be defined, from now on, as \(\mathbb{D}_{X_{i}}\). The parents of a decision node \(Pa(D)\) (\(C\) or other \(D\)) contain the information set available (observable) for \(D\) to decide; utility nodes \(U\) usually have no children and are probabilistically determined by their parents \(Pa(U)\) (\(C\) or \(D\)) [31]; and its DAG and properties offer advantages regarding complexity and explainability regarding the traditional Extensive Form Game representations [34].

4.2 Problem and model definition↩︎

This work intends to recommend minimal SPCSFs to fulfil the requirements indicating the degree of coverage of the 7 considered security dimensions. To that end, a MAID \(\mathcal{M}\) models the security dimensions as agents \(A\), encodes SPCSFs’ coverage as chance nodes \(C\), uses SPCSFs’ domain in the decision nodes \(D\) and formally defines the utility function for agents \(A\) as the utility nodes \(U\). Chance nodes internally encode each SPCSF’s SW-implementability and score from \(DS_{8}\); whereas utility nodes embody the utility formulation for each agent, formalising its behaviour from its relation to other agents (which can tend from collaboration to competition) and the contribution received from each SPCSF, among others. Edges between these represent how much a SPCSF contributes to cover an agent. Fig. 6 depicts a simple MAID representing C, D and U nodes as circles, squares and diamonds; respectively. Dotted edges denote the information links or information available to an agent/decision (e.g. \(Pa(D) \rightarrow D\)), whereas solid edges indicate the dependency over other nodes into the calculation of the utility or payoff of the agent.

None

Two modes \(M\) of MAID are proposed to model this problem: Fig. 7 shows the full MAID and Fig. 8 depicts the clustered MAID. These have \(|V| = 111\), \(|E| = 203\) and \(|V| = 25\), \(|E| = 39\) vertices and edges, respectively. Edges \(E\) connect \(C\) with \(D\) and \(D\) with \(U\). Vertices \(V\) directly represent (full \(M\)) or indirectly aggregate (clustered \(M\)) the 97 SW-implementable SPCSF from the dataset, along with 7 decision and utility nodes to represent the agents. Each \(C\) node is coloured based on the \(D\) and \(U\) nodes it contributes and relates to. Each proposed MAID \(\mathcal{M}\) is used for a multi-player, non-zero-sum, simultaneous game with utilities that consider both incentives and penalties. \(\mathcal{M}\) can be used to find suitable joint, optimal cross-agent policy or decision profile \(\sigma^{*}\) that cannot be improved by any agent’s unilateral deviation; which acts as an approximate equilibrium.

It is important to note that, while obtaining exact optimal equilibria for a full MAID is intractable using exact calculations methods, the clustered MAID heavily brings the computational complexity down; making it still possible to obtain an approximate equilibrium. Both \(M\) are solved in this work with no-regret learning dynamics rather than exact calculations to ensure proper scalability; specifically using the Hedge / Exponential Weights Algorithm (EWA) [36] and the Exponential-weight algorithm for Exploration and Exploitation (EXP3) [37] methods. Section 4.5 explains the rationale for this approach, including empirical validation and theoretical complexity analysis.

4.3 Utility definition↩︎

The utility function is constructed on-the-fly during the MAID generation from the dataset \(DS_{8}\). The utility \(U_{a}(\sigma)\) for an agent \(a\) evaluated for a joint decision profile \(\sigma\) is defined in Eq. 2 .

\[\label{eq:implementation-model:utility} U_{a}(\sigma) = \underbrace{\Big(S_{a}(\sigma) + Q_{a}(\sigma)\Big)}_{\text{Incentives (}\Delta_{a}(\sigma)\text{)}} - \underbrace{\Big(P^{p}_{a}(\sigma) + P^{s}_{a}(\sigma)\Big)}_{\text{Penalties (}\Gamma_{a}(\sigma)\text{)}}\tag{2}\]

The utility encodes the motivation (objective) per agent \(a\), which is to be maximised as per Eq. 4 . To do so, it evaluates \(U_{a}\) to determine its optimal strategy \(\sigma_{a}^*\) against the strategies of other agents \(\sigma_{-a}\). The objective of the system, in Eq. 5 , is to find the optimal joint strategy (discrete) profile \(\sigma^*\) or approximate equilibrium; where incentives and penalties are balanced across agents.

\[\tag{3} \begin{align} \sigma_{a}^{*} &= \mathop{\mathrm{arg\,max}}_{\sigma_{a} \in \mathbb{D}} ~ U_{a}(\sigma_{a}, \sigma_{-a}) \tag{4} \\[0.5em] \sigma^* &= \{ (a, \sigma_{a}^*) \mid a \in A \} \tag{5} \end{align}\]

In practice, \(U_{a}\) must be able to guide the agent \(a \in A\) to explore the minimal number of SPCSFs that can best serve each security requirement \(\tau_{a}\) per agent \(a\) and maximise tailored coverage for its associated security dimension. It considers contributions providing both incentives \(\Delta_{a}(\sigma)\) and penalties \(\Gamma_{a}(\sigma)\), as per Eq. 6 . The incentives group (i) the synergy \(S_{a}(\sigma)\) between any pair of agents; and (ii) the overall quality \(Q_{a}(\sigma)\) for a chance node in the MAID — where chance nodes are either a SPCSF cluster or item, depending on the type of MAID \(\mathcal{M}\), and prioritising controls with higher coverage towards agents. The penalties group (iii) the provisioning penalty \(P^{p}_{a}(\sigma)\); and (iv) the sparsity penalty \(P^{s}_{a}(\sigma)\) to maximise accuracy and minimise the number of SPCSFs in use, respectively.

\[\tag{6} \begin{align} S_{a}(\sigma) &= \sum_{-a \in A \setminus \{a\}} C_{-a}(\sigma) \cdot \rho_{a,-a} \cdot w_s \tag{7} \\[0.5em] Q_{a}(\sigma) &= \sum_{\omega \in \Omega} p(\omega, \sigma) \cdot q(\omega, a) \tag{8} \\[0.5em] P^{p}_{a}(\sigma) &= \lambda(E_{a}, \tau_{a}) \cdot E_{a}^2 \tag{9} \\[0.5em] P^{s}_{a}(\sigma) &= \sum_{\omega \in \Omega} p(\omega, \sigma) \cdot \frac{P^{s}_{max}}{|\Omega|} \tag{10} \end{align}\]

As a scaling factor in \(S_{a}(\sigma)\) via \(C_{a}(\sigma)\) in Eq. 11 , \(Q_{a}(\sigma)\), and \(P^{s}_{a}(\sigma)\); the weight \(p(\omega, \sigma) \in [0, 1]\) indicates how likely each SPCSF \(\omega\) is to contribute to cover a given agent under the examined joint policy profile \(\sigma\). That is, it represents the need of an SPCSF to satisfy each requirement. It is worth noting that this weight is obtained deterministically and does not follow a known probability distribution. In particular, Algorithm 10 maps the agents’ possible choices from \(\mathbb{D}_{A}\) to numerical requirement weights denoting the required coverage of each agent’s associated security dimension. \(p(\omega, \sigma)\) takes the maximum weight across all \(a \in A\) covered by each SPCSF \(\omega\). Besides this, other weights defined for the incentive (\(w_{s}\)) and penalty (\(W_{u}, \theta_{neg}, P^s_{max}\)) utility terms were fine-tuned after manual sensitivity testing. It had been observed that \(w_{s} > 0.1\) hoarded SPCSFs to increase synergy at the cost of over-provisioning; that small \(W_{u}\) were not penalising enough the under-provisioning and were increased orders of magnitude; that \(\theta_{neg}\) should only consider for negative correlation under a minimal threshold (analysed from Fig. 9) and that \(P^s_{max}\) was calculated to be a bit larger than the value delivered by \(S_{a}(\sigma)\), yet considerably smaller than that given by \(P^{p}_{a}(\sigma)\). Finally, the \(C_{max}\) is used in \(S_{a}(\sigma)\) as the upper boundary to limit coverage when aggregating contributions.

4.3.1 Incentive \(S_{a}(\sigma)\): dimension synergy↩︎

The synergy across agents \(S_{a}(\sigma)\), in Eq. 7 , rewards agent \(a\) when the selected SPCSFs provide simultaneous coverage to any other agent \(-a \in A \setminus \{a\}\). This models the nature of collaborative vs competitive coverage; given that when covering an agent \(a\), the coverage of another agent \(-a\) can be either augmented or diminished. This is obtained from the correlation matrix \(R\) across agents and fosters complementary coverage relationships, measured through the cross-dimensional coverage \(C_{-a}(\sigma)\) and the dimensional correlation \(\rho_{d,-a}\). The total synergy is scaled by weight \(w_s\) (here, 0.1) to avoid dominating the utility function that guides the exploration.

\[\label{eq:implementation-model:utility:coverage} C_{a}(\sigma) = \min\left(C_{max}, \sum_{\omega \in \Omega} p(\omega, \sigma) \cdot v_{\omega, a}\right)\tag{11}\]

Eq. 11 denotes the cross-dimensional coverage \(C_{a}(\sigma)\) per dimension \(a \in A\). It is worth noting that \(C_{-a}(\sigma)\) is derived by evaluating \(C_{a}(\sigma)\) identically, yet for other dimensions \(-a\). \(C_{a}(\sigma)\) aggregates the contribution \(v_{\omega, a} \in [0,1]\) of SPCSF (chance nodes \(\mathcal{N}_\omega\), or simply \(\omega \in \Omega\) in \(\mathcal{M}\)), towards each agent \(a\); which is multiplied by the weight \(p(\omega, \sigma)\). In order to account for multiple SPCSFs covering a given \(a\) (even exceeding 100% coverage), the upper bound is set to \(C_{max}\) (here, 1). To compute the final synergy \(S_{a}(\sigma)\), the dimensional correlation \(\rho_{a,-a} \in [-1, 1]\) expresses the synergy between dimensions \(a\) and \(-a\), as obtained from \(R\). Fig. 9 provides an example of \(R\). \(\rho_{a,-a}\) acts as a weight: (i) if positive, agents are complementary and it yields an incentive for SPCSFs covering also other dimensions; and (ii) otherwise, the agents are in conflict; likely hindering coverage of the current agent, and thus discourages the selection of such SPCSFs.

4.3.2 Incentive \(Q_{a}(\sigma)\): SPCSF quality↩︎

The quality incentive \(Q_{a}(\sigma)\), in Eq. 8 , rewards an agent \(a\) that explores SPCSFs with high alignment on both the security baseline and its total score; where the baseline alignment is favoured. This can act as a tie breaker when multiple SPCSFs offer equivalent coverage. To obtain this value, it considers the weight \(p(\omega, \sigma)\) and multiplies it to the evaluation function \(q(\omega, a)\) - which combines a weighted sum of (i) the NIST SP 800-53 baseline, aligning its significance to that defined in the joint profile \(\sigma\) for \(a\); and (ii) the total score; both available in \(DS_{8}\) for each SPCSF.

4.3.3 Penalty \(P^{p}_{a}(\sigma)\): under- and over-provisioning↩︎

The recommendation of suitable SPCSFs to meet the security requirements \(\tau_{a}\) per agent \(a\) can be seen as a resource provisioning problem (i.e. provisioning SPCSFs to cover requirements on security dimensions). This can incur two undesirable situations: under-provisioning (recommending fewer SPCSFs than needed) and over-provisioning (recommending redundant SPCSFs). From a conservative cybersecurity provisioning approach, under-provisioning increases the vulnerability of the environment; whereas over-provisioning wastes resources (e.g. time or adoption effort) - making the latter preferable to an exposed attack surface. The provisioning penalty \(P^{p}_{a}\), in Eq. 9 , is introduced to discourage agents from exploring the decision space in ways that can more easily lead to such conditions. To identify both conditions, the difference (distance) between the provided coverage \(C_{a}(\sigma)\) and requested coverage \(\tau_{a}\) of an agent \(a\) is calculated. This is captured by the error \(E_{a} = C_{a}(\sigma) - \tau_{a}\), with a negative value indicating under-provisioning; and otherwise, over-provisioning. The squared error \(E_{a}^{2}\) is multiplied by a dynamic weight function \(\lambda(E_{a}, \tau_{a})\) that defines a proper coefficient depending on conditions of under-provisioning (\(E_{a} < 0\)) or over-provisioning (\(E_{a} \geq 0\)). In the first case, it uses \(W_{u}\) (here, 50) to heavily penalise this case; whereas in the second case it uses \(W_{o}\), obtained from the correlation matrix \(R\) across agents and from a multiplier proportional to the maximum over-provisioning (i.e. exceeding coverage regarding security requirements), which is bounded between \(0.1\) and \(1.0\).

\[\label{eq:implementation-model:utility:term:penalty:overprovision} \begin{gather} \mathcal{C}_{neg} = \{ \rho_{i,j} \in R_{tri} \mid \rho_{i,j} < \theta_{neg} \} \\[0.5em] V = \begin{cases} \frac{\min(\mathcal{C}_{neg}) + \text{mean}(\mathcal{C}_{neg})}{2} & \text{if } \mathcal{C}_{neg} \neq \emptyset \\ 0 & \text{otherwise} \end{cases} \\[0.5em] W_{o} = \max\Big(w_{min}, \min\big(w_{max}, w_{base} + \gamma \cdot V\big)\Big) \end{gather}\tag{12}\]

\(W_{o}\), in Eq. 12 , is dynamically calculated to avoid agents from getting a direct penalty when the available SPCSFs in the dataset lead to over-provisioning. It is calculated dynamically from the conflicting dimensions \(\mathcal{C}_{neg}\), using the upper triangular elements \(R_{tri}\) of the security dimensions’ correlation matrix \(R\) whose negative correlation is below a negative correlation threshold \(\theta_{neg}\) (here, -0.05). It is worth noting that \(R\) is also used to calculate \(S_{a}\). From that, \(V\) is calculated from the minimal and average values of these conflicts. Then, \(W_{o}\) increases \(V\) by \(\gamma\) and keeps it bound between \(w_{min}\) and \(w_{max}\) around a baseline \(w_{base}\). In this case, the chosen values for these parameters were 2, 0.5, 2 and 2, respectively. Negative correlations (seen as conflicts between agents) incur less over-provisioning; and thus, \(W_{o} \propto 1 / V_{i}\). In these cases, a higher \(V\) denotes that covering multiple dimensions becomes more complicated and likely require more overlapping SPCSFs; thus \(W_{o}\) must be more lenient.

4.3.4 Penalty \(P^{s}_{a}(\sigma)\): sparsity↩︎

The sparsity penalty \(P^s_{a}(\sigma)\), in Eq. 10 , applies L1 regularisation to limit feature selection. That is, it penalises the selection of unnecessary SPCSFs and instead fosters the selection of a smaller, more meaningful set of SPCSFs that can equally cover each security dimension \(a\). To do so, it uses a global penalty \(P^s_{max}\) (here, 2) as the upper boundary evenly across the number of available SPCSFs (\(|\Omega|\)). With a smaller number of SPCSFs (e.g. clustered mode), this penalty increases considerably — since adding a cluster may incur massive over-provisioning depending on the SPCSFs it contains. Conversely, with a larger number of SPCSFs (e.g. in the full mode), this penalty is reduced to allow selecting more specialised SPCSFs (i.e. those covering less dimensions at the time). This global penalty is then weighted by \(p(\omega, \sigma)\). Therefore, \(P^{s}_{a}(\sigma)\) can adapt to the topology of the MAID \(\mathcal{M}\) (and thus, to the selected mode \(M\)) and the maximum available SPCSFs; guiding towards more accurate decision profiles.

4.4 Recommendation procedure↩︎

The MAID model \(\mathcal{M}\) is constructed along with the utility function \(U_{a}\) per agent \(a\) (Algorithm 10). This is used for exploring the decision space and recommending SPCSFs (Algorithm 11). Notation is available in Table 5.

4.4.1 MAID construction from dataset↩︎

Algorithm 10 uses the dataset \(DS_{8}\) from Algorithm 3 along with the dataset filtering threshold \(T\), the mode \(M\) (i.e. clustered or full) used to represent the data in the MAID, the maximum clusters \(G_{c}\) (if needed), the data for SPCSFs \(\Omega\), the list of agents \(A\) and their domain \(\mathbb{D}_{A}\), the correlation matrix \(R\) across agents and the requirements per agent \(\vec{\tau}\).

First, the dataset \(DS_{8}\) is filtered in Algorithm 10, line [alg:implementation-model:maid-generation:line:s1]. This filters SPCSFs according to their total score through a configurable threshold \(T\), and ends up with a subset of SPCSFs \(DS_{8f}\) that will be used to cover the security requirements \(\vec{\tau}\). Depending on \(M\), the set of available, filtered SPCSFs to be used as chance nodes in the MAID \(\mathcal{M}\) is further modified, as \(\Omega\). If the mode \(M\) is clustered, the GMM clustering process creates clusters containing specific SPCSFs according to their coverage of the agents. This process can either use an optional upper limit for clusters (\(G_{c}\)) or apply the Bayesian Information Criterion (BIC) to find the optional number of clusters. \(G_{c}\) is optional and the default runs GMM with BIC to automatically find this value. Otherwise, for a full MAID, the whole \(DS_{8f}\) is used. This runs in Algorithm 10, lines [alg:implementation-model:maid-generation:line:s2]-[alg:implementation-model:maid-generation:line:s5].

With this data, the vertices \(V\) and edges \(E\) for the MAID \(\mathcal{M}\) can be extracted from the filtered SPCSFs in \(\Omega\). \(V\) is defined from the different nodes \(\mathcal{N}_c\), \(\mathcal{N}_d\) and \(\mathcal{N}_u\). These nodes encode some information as well: chance nodes \(\mathcal{N}_c\) comprise its approximate, average total score (Algorithm 10, lines [alg:implementation-model:maid-generation:line:s6]-[alg:implementation-model:maid-generation:line:s7]); decision nodes \(\mathcal{N}_d\) host the domain \(\mathbb{D}\), or values that can be considered (Algorithm 10, line [alg:implementation-model:maid-generation:line:s9]); and utility nodes \(\mathcal{N}_u\) contain hybrid utility functions \(U_{A}\) (Algorithm 10, lines [alg:implementation-model:maid-generation:line:s12]-[alg:implementation-model:maid-generation:line:s18]). It must be noted that chance nodes use here a probabilistic normal distribution around its average total score (\(\mu\)) and with a deviation (\(\sigma\)) of 0.1, restricted to the \([0.0, 1.0]\) interval. These represent the uncertainty over exact values to accommodate changes in the origin data, whether from \(DS_8\) or its sourced datasets (e.g. changes in 1-2 columns for \(DS_{7}.cm\) can get \(\sim 10\%\) average variability), and introduce variability to test the framework under varying conditions. Utility functions base on direct or indirect data from the dataset, e.g. the coverage \(C_{a}\) of an agent (Algorithm 10, line [alg:implementation-model:maid-generation:line:s11]). The edges \(E_{A}\) (constructed in Algorithm 10, line [alg:implementation-model:maid-generation:line:s10]) express the flow of information by connecting chance nodes to decision nodes, and decision nodes to utility nodes. Finally, a MAID \(\mathcal{M}\) can now be generated for each set of parameters (e.g. mode \(M\)) in Algorithm 10, line [alg:implementation-model:maid-generation:line:s19].

4.4.2 Game-theoretic exploration and recommendation↩︎

Once MAID \(\mathcal{M}\) is constructed, Algorithm 11 runs two stages: (i) the game-theoretic exploration across the decision space for all agents, invoking each utility action \(U_{a}\) to maximise each agent’s payoff and obtain the most accurate joint solution or (discrete) profile \(\sigma^{*}\) that can meet best the security requirements without exceeding these; and (ii) the recommendation stage itself, which navigates the sorted SPCSFs and select those that contribute to the coverage without exceeding it or exhausting the budget.

Based on the MAID \(\mathcal{M}\), mode \(M\) and learning algorithm \(\mathcal{L}\), the initial learning parameters are defined (Algorithm 11, lines [alg:implementation-model:gt-exploration-recommendation:line:s1]-[alg:implementation-model:gt-exploration-recommendation:line:s2]). The probability weights \(W\) per agent and the number of iterations \(L\) per algorithm \(\mathcal{L}\) (i.e. Hedge/EWA and EXP3) are calculated, along with the empty history set \(\mathcal{H}\) to track explored decisions.

For each iteration \(l \in L\), the learning algorithm picks, in a stochastic manner, a joint strategy profile \(\sigma^{(l)}\) with values from domain \(\mathbb{D}_{A}\) and a probabilistic distribution \(\vec{p}\) (Algorithm 11, line [alg:implementation-model:gt-exploration-recommendation:line:s4]). This profile represents the collective explored/covered agents at that iteration. This is recorded as part of the decisions’ history (Algorithm 11, line [alg:implementation-model:gt-exploration-recommendation:line:s5]). Then, the utility function \(U_{a}\) is invoked per agent considering the current profile \(\sigma^{(l)}\); and its result is used to update the probability weights \(W\) and sampling distribution \(\vec{p}\), in order to steer the stochastic selection of the profile (Algorithm 11, lines [alg:implementation-model:gt-exploration-recommendation:line:s6]-[alg:implementation-model:gt-exploration-recommendation:line:s8]). Finally, and after running all \(L\) iterations per algorithm \(\mathcal{L}\); the history of explored decisions \(\mathcal{H}\) is analysed to extract the most accurate profile \(\sigma^*\) from it (Algorithm 11, line [alg:implementation-model:gt-exploration-recommendation:line:s9]). Such profile \(\sigma^*\) minimises the distance between the required \(\tau_{a}\) and obtained \(c_{a}\) degree of coverage to an agent \(a\); where this coverage is obtained from the joint profiles previously explored and tracked in \(\mathcal{H}\). With this, the game-theoretic exploration stage produces the coverage profile \(\sigma^*\), implemented as a discretised vector that maps each agent \(a\) to a degree of coverage \(c_{a}\) that can be low, intermediate or high (or none), depending on the amount of coverage. Since this result is met after running the utilities \(U_{a}\), as the approximate equilibrium for the agents it indicates which security requirements can be effectively met and to which extent.

From this point onwards, the recommender stage takes place. An initial ranking stage (Algorithm 11, lines [alg:implementation-model:gt-exploration-recommendation:line:s10]-[alg:implementation-model:gt-exploration-recommendation:line:s14]) evaluates how well each SPCSF \(\omega \in \Omega\) can cover the requested security requirements. This is calculated by the identified similarity \(f_\omega\) between the coverage provided by such SPCSF \(\omega\) and the expected total coverage \(\vec{\tau}\) (Algorithm 11, line [alg:implementation-model:gt-exploration-recommendation:line:s12]) and is used to discourage both under- and over- provisioning. Then, all items added to \(F\) are ranked from most to least similarity, yielding \(F_{s}\).

After the ranking, the second part of the recommendation stage ensures that the coverage provided by the approximate equilibrium \(\sigma^{*}\) acts as an upper bound; avoiding over-provisioning and allowing to identify the subset of the SPCSFs \(\mathcal{R}\) to recommend. The set \(\mathcal{R}\) is initialised along with the coverage vector \(\vec{c}\). The best joint profile \(\omega^*\) is considered, along with the passed security requirements \(\vec{\tau}\), to determine the upper boundary \(\vec{\beta}\) for coverage that each agent \(a\) must achieve to avoid over-provisioning (Algorithm 11, lines [alg:implementation-model:gt-exploration-recommendation:line:s15]-[alg:implementation-model:gt-exploration-recommendation:line:s16]). With all of that, the recommender runs a loop to accumulate adequate SPCSFs in Algorithm 11, lines [alg:implementation-model:gt-exploration-recommendation:line:s17]-[alg:implementation-model:gt-exploration-recommendation:line:s20]. This process checks whether (i) the requested security requirements \(\vec{\tau}\) are still unmet by the current achieved coverage \(\vec{c}\), while ensuring that (ii) the current amount of recommendations do not exceed the internally defined budget \(B\); it continues taking SPCSFs from those available at the sorted pool \(F_{s}\).

\[\tag{13} \begin{align} \Delta\,\mathrm{cov}(\omega, \vec{c}) &= \sum_{a \in A} \Big( \min(c_{a} + v_{\omega, a}, \tau_{a}) - c_{a} \Big) \tag{14} \\[0.5em] \mathrm{pen}_{o}(\omega, \vec{\beta}, W_o) &= W_o \cdot \sum_{a \in A} \max\big(0, (c_{a} + v_{\omega, a}) - \beta_{a}\big) \tag{15} \end{align}\]

At each iteration, the best candidate SPCSF \(\omega^{*}\) is selected, trying to maximise only the useful coverage for requested agents \(\Delta \mathrm{ cov}\) and to minimise unnecessary coverage \(\mathrm{pen}_{o}\), in the same way as the under- and over- provision constraints in the utility function itself. These conditions are detailed in Eq. 14 and Eq. 15 respectively; with \(c_{a}\) being the accumulated coverage per agent \(a\) and \(v_{\omega, a}\) being the coverage of SPCSF \(\omega\) to agent \(a\). It is worth noting that, if an agent (its represented security dimension) is already fully covered, \(\Delta\,\mathrm{cov}\) is zero. Then, the recommended set \(\mathcal{R}\) and the coverage are updated adequately with the best SPCSF \(\omega^{*}\). In a final pass (Algorithm 11, lines [alg:implementation-model:gt-exploration-recommendation:line:s21]-[alg:implementation-model:gt-exploration-recommendation:line:s23]), \(\mathcal{R}\) is reviewed to ensure the bare minimum SPCSF set is proposed to cover the security requirements \(\vec{\tau}\).

Results for an exploration and recommendation stage (Algorithm 11) using a selected requirements profile chosen at random are shown in Fig. 12, as evaluated for both full (Fig. 7) and clustered (Fig. 8) modes and applying two different learning algorithms \(\mathcal{L}\) to minimise the regret over the selected decisions: Hedge Learning and EXP3. In the radar plot, each agent is located in a vertex and coloured differently. The explored space is identified by dots, with arrows indicating the direction of the exploration considering the best joint profile (approximate equilibrium). The security requirements \(\vec{\tau}\) are shown in a bold dashed polygon, whose shape is the approximation to the ideal, exact fit that indicates no under- and over-provisioning. The best joint discrete profile (approximate equilibrium) obtained from the exploration stage is shaded in light blue and its contour is dotted. The crosses at each agent denote its perimeter or boundaries. Finally, the polygon in purple shade and solid contour reflects the degree of coverage for the agents suggested in the final, recommending stage.

The best joint profile is selected out of the history of explored profiles \(\mathcal{H}\), where each learning algorithm \(\mathcal{L}\) runs different iterations \(L\), suited to their behaviour. As in the utility function, the criteria to select it heavily penalises under-provisioning, and in case of similar coverage between two decision profiles, it favours those that minimise over-provisioning. This can be observed in the blue shades, which are always larger than the purple shades. The purple shade will more often than not have minor over-provisioning to ensure covering at least the desired degree of security. Conversely, it also can incur under-provisioning if the SPCSF dataset (\(DS_8\)) is too small (i.e. a high threshold \(T\)); which is considerably more problematic. In the execution shown in Fig. 12, the recommendation between the clustered and full MAID returned slightly different SPCSFs. It is worth noting that no under-provisioning is found in that execution until filtering the dataset with \(T = 0.4\) (i.e. total score of at least 0.4). Here, the accountability agent is penalised due to related SPCSFs being filtered out due to its lower total score; considerably reducing the original 16 identified SW-implementable SPCSFs from Table 3. This is comparable to executions with other requirements (Section 5.3).

4.5 Complexity limitations↩︎

Finding exact equilibria faces two primary computational limitations: (i) generating and storing CPDs and (ii) evaluating the MAID subgames across a large decision space. In fact, these make exact calculation intractable for models requiring extensive data combinations, such as the full MAID (Fig. 7). More compact models, such as the clustered MAID (Fig. 8) alleviate these limitations and allow computing some exact equilibria, yet very slowly. This section explains these findings and the motivation for no-regret online learning dynamics, which allow to bypass the intractability and obtain an approximate equilibrium considerably faster.

4.5.1 Space: storing probabilistic data↩︎

In BN and derived approaches using exact inference, the conditioned Bayesian probabilities per decision node are computed based on its parent chance nodes, as per \(\mathbb{P}(D \mid Pa(D))\), and used to populate its CPDs. In the full MAID \(\mathcal{M}\), in Fig. 7, the in-degree (number of parents) of decision nodes ranges from 2 to 67. Eq. 17 and 18 determine the upper bound for the number of CPD entries \(\mathcal{C}_{e}\) and the associated memory size \(\mathcal{C}_{s}\), respectively. These scale exponentially with \(m_p = |max(Pa(D))|\), but also grow with \(\mathbb{D}_{V}\) — namely, the maximum number of parents and decision domain per decision node.

\[\tag{16} \begin{align} \mathcal{C}_{e} &\le |V| \cdot |\mathbb{D}_{V}|^{m_{p} } \mid m_{p} = \text{max(}|Pa(V)|\text{)} \tag{17} \\[0.5em] \mathcal{C}_{s} &= \frac{4}{{10^{9}}} \cdot \sum_{v_=1}^{|V|} (|k|+|\mathbb{D}_{v}|-1) \cdot (|\mathbb{D}_{k}|^{|k|}) \mid k = Pa(v) \tag{18} \end{align}\]

4pt

Table 6 summarises the application of Eq. 17 and Eq. 18 to \(\mathcal{M}\) based on the iteratively filtered dataset \(DS_{8}\) based on score constraints and the SW-implementation criteria, which reduces \(m_{p}\). It shows the vertices \(|V|\) and edges \(|E|\) per \(\mathcal{M}\) generated from each filtered dataset — where \(|V| = N + 14\) to accommodate the decision and utility nodes per security dimension. It also finds the \(\mathcal{C}_{e}\), equivalent to the minimum independent parameters required to represent all joint CPDs [17]; and the size in GBs in \(\mathcal{C}_{s}\). \(\mathcal{C}_{e}\) assumes \(|\mathbb{D}_{V}| = 10\), since agents can decide on values from the score deciles (Fig. 4). In the best case (i.e. \(DS_{8}\) reduced to half after filtering it with threshold \(T=0.4\)), \(\mathcal{C}_{e}\) is still around the order of \(10^{33}\) entries. Even if also reducing decisions to \(|\mathbb{D}_{V}| = 4\), in the best case \(\mathcal{C}_{e}\) would yield around the order of \(10^{20}\) entries. Thus, finding the exact equilibrium is intractable for the full MAID. Since exact inference on BN models is NP-hard [38], [39], processing all entries for such a dense decision or policy space incurs a combinatorial explosion that requires prohibitive amounts of memory. One way of bringing the complexity of the MAID considerably down is by decreasing the in-degree of decision nodes \(D\); e.g. applying (child-friendly) parent divorcing [40] or grouping chance nodes (SPCSFs) together, e.g. identifying latent variables [41] or clustering based on similar contribution from SPCSFs towards agents. The clustering approach is examined here applying Gaussian Mixture Model (GMM) along with the Bayesian Information Criterion (BIC), to perform soft clustering and group SPCSFs with similar properties into latent variables (clusters). The resulting clusters group chance nodes into \(k=11\) groups, as shown in Fig. 8. This drastically reduces \(D\)’s in-degree and the number of edges, limiting CPDs to a computationally tractable size. \(\mathcal{C}_{e}\) would reduce in the order of \(10^9\) and \(10^5\) for \(|\mathbb{D}_{V}|\) equal to 10 and 4, respectively.

4.5.2 Computation: exploring the decision space↩︎

Besides memory limitations, a game-theoretic system must evaluate expected utilities \(EU_d(\sigma)\) across the agents’ joint strategy profiles \(\sigma\) to find optimal equilibria \(\sigma^{*}\). Table 7 summarises the upper complexity bounds to converge on such optimal outcome in terms of (i) search space, understood as the (joint or individual) profiles \(\sigma\); and (ii) of time complexity across these methods (iterations). Notation refers to the number of agents \(|A|\), decision domain per agent \(\mathbb{D}_{A}\), number of iterations \(L\), maximum number of parents (in-degree) in the decision \(D\) nodes from the MAID \(\mathcal{M}\) as \(m_{p} = \text{max(}|Pa(V)|\text{)}\) and \(s\) being the worst-case exponent used by the implemented LP solver.

Finding an exact Nash Equilibrium [49] is PPAD-complete, even for 2-player games [27], and its complexity scales exponentially with \(|A|\). More efficient approaches exist for graphical game models like MAIDs [50] that exploit conditional independence [31] and can bring complexity down to polynomial time when bound to \(|\mathbb{D}_{A}| = 2\) and \(|A| = 3\) [42] and exploiting the relevance graph [51]; yet increasing agents or domain is also intractable on commodity hardware. Correlated Equilibrium (CE) [52] and Coarse Correlated Equilibrium (CCE) [44] offer more computationally tractable alternatives for multi-agent games. CE can be solved via Linear Programming (LP) [43], [53], but that requires generating utility matrices that scale exponentially with \(|A|\) and can still be PPAD-complete under certain constraints [54]. Best-Response Dynamics (BRD) is Polynomial Local Search (PLS)-complete [45] with exponential worst-case time, like Nash. Min-Max Regret (MMR) [46] and Fictitious Play (FP) [47] require significant memory to persist large utility matrices as in LP implementations (MMR) or a potentially large number of historical probabilistic actions (FP).

To overcome time and memory complexities, no-regret learning dynamics work in a decentralised approach that allows each agent to independently focus on minimising its cumulative regret (i.e. a poor action taken from \(\mathbb{D}_{A}\) that diverges from the best possible one) through \(L\) iterations. These can ensure, over time, that agents obtain an average reward that is asymptotically as good as that of any fixed strategy from the explored decision space. This mechanism does not require loading the full decision strategy space into memory, which can converge rapidly to CCE [55] and present upper boundaries on memory and time complexities that are considerably more bounded than other methods. The convergence of these methods towards optimal strategies is determined by the upper bounds for regrets \(O(\sqrt{T \cdot \ln N})\) and \(O(\sqrt{T \cdot N \cdot \ln N})\), respectively; with \(T = L\) as the number of iterations and \(N = |\mathbb{D}_{A}|\) as the available actions/strategies per agent. The selection of learning algorithms depends on factors like the comprehensiveness of the MAID \(\mathcal{M}\) or the assured convergence of each algorithm. No-regret learning algorithms are very well-suited for this model, given its reduced complexity and fast convergence to CCE.

5 Evaluation↩︎

The Security DSS was evaluated in two ways: according to its internal behaviour and against a baseline. On the one hand, its internal behaviour used different game-theoretic and recommender system’s metrics. The metrics used for the full evaluation are described and interpreted first in Section 5.2. A use case is proposed in Section 5.3 to illustrate specific requirements passed to the DSS and analyse its outcomes (Table 8) and aggregated statistical values for a subset of dataset sizes (Table [tbl:tab:evaluation:use-case]). On the other hand, the two baseline implementations were compared against the proposed work in Section 5.4. It shows first a summarised evaluation, aggregating data across a number of dataset sizes, modes and learning algorithms in Table 10. Full details for the internal evaluation of the proposed recommender and the comparison against the greedy, filtered baseline are available in Table 11 and Table 12, respectively. Both tables summarise statistical values across different filtering iterations of the dataset (using thresholds \(T\)) for each metric, where each cell shows their mean ± SD.

It is important to note that the threshold \(T\) (between 0.0 and 0.8) directly influences the size of dataset \(DS_{8}\), since it filters those \(N\) SPCSFs whose score_total is equal or higher than \(T\). A bigger threshold yields fewer SPCSFs (\(T \propto 1 / N\)). Consequently, a good recommending accuracy expects a sufficiently large \(N\), as better SPCSFs can be recommended from a larger pool.

5.1 Methodology↩︎

The following metrics are considered: (i) performance (time taken to produce the recommendation); (ii) efficiency (minimisation of resources used to cover the requirements); (iii) accuracy (recommendation fitting with respect to requirements); and (iv) equity (degree of satisfaction per requirement without actively penalising others). The recommender was run \(m \cdot n \cdot o\) times in nested iterations for both validations; with (i) \(m\) being the range of thresholds (\(T\)); (ii) \(n\) being the number of iterations (100); and (iii) \(o\) each of the modes (\(M\)), i.e. "clustered" or "full". The security requirements are randomly generated within the \(n\) loop to ensure different constraints considered in the recommendation, which also introduces considerable variability in multiple metrics.

The proposed Security DSS (comprised by Algorithms 10 and 11) explores the decision space within each mode \(M\) in the \(o\) loop by using two learning algorithms \(\mathcal{L}\) (Hedge / EWA and EXP3) to minimise the regret on the selected choice. These repeat explorations for 200 and 400 iterations, respectively. This work is also compared in Section 5.4 against a greedy filtered recommender that implements the last sorting and filtering stage from Algorithm 11 (lines [alg:implementation-model:gt-exploration-recommendation:line:s15]-[alg:implementation-model:gt-exploration-recommendation:line:s23]).

5.2 Internal evaluation↩︎

The box plots in the following subsections render average values and represent the IQR.

5.2.1 Performance↩︎

The performance of the system evaluates the time taken by the different stages from the Security DSS. That is, to (i) generate the MAID first; (ii) explore the decision space to look for a joint decision profile as approximate equilibrium; and (iii) perform the recommendation filtering and ranking.

Times are shown in Table 11. Both the time to generate the MAID \(\mathcal{M}\) and the recommendations \(\mathcal{R}\) are negligible in comparison with the learning time \(\mathcal{L}\) for the game-theoretic exploration of the space; which is mainly caused by the hundreds of iterations required to reach enough decision profiles. The MAID generation time is considerably slower for the clustered mode with respect to the full mode; as expected given that the clustered mode always run the GMM clustering process. This difference in time reduces with a higher \(T\) and less available SPCSFs, as there are less operations to perform. The recommendation time for the full mode, on the other hand, approximately doubles that of the clustered mode; given the larger number of chance nodes \(\omega\) it handles directly. This also decreases along with fewer SPCSFs.

5.2.2 Efficiency↩︎

The efficiency of the system is measured by (i) the Price of Anarchy (PoA) from Eq. 19 , which measures the efficiency of the game-theoretic equilibrium; as well as the (ii) minimum number of SPCSFs, which heavily depends on the dataset size and is also limited by an internal budget control to avoid recommending unnecessary controls when security requirements are covered – and thus limit over-provisioning.

\[\label{eq:evaluation:internal:poa} \text{PoA} = \frac{\max_{\sigma \in \mathcal{H}} \text{Welfare}(\sigma)}{\text{Welfare}(\sigma^*)}\tag{19}\]

The PoA checks how the system efficiency degrades with selfish behaviour from the agents through the ratio between the optimal solution (maximum welfare of all explored strategies \(\sigma \in \mathcal{H}\)) and the achieved equilibrium (welfare of the optimal joint profile \(\sigma^{*}\)). The perfect ratio, and minimum boundary, is 1. Fig. 13 showcases the distribution of the PoA values across different dataset sizes; and averaged values are provided in Table 11. There, PoA stays equal or very close to 1 for both learning algorithms in the clustered mode; and grows with the full (large) MAID for both learning algorithms until \(T = 0.5\), especially increased for EXP3 and suffering from spikes in some outliers.

On the other hand, the minimum number of SPCSFs is computed to stop recommending SPCSFs the moment it satisfies the requirements, as well as re-iterating to minimise unwanted over-provisioning; while ensuring there is no avoidable under-provisioning. Fig. 14 plots the number of recommended SPCSFs with different T (i.e. dataset sizes). A higher number of SPCSFs allows the Security DSS to recommend more of them to better fit the security requirements; yet that amount must be limited to controls that strictly cover these requirements, without adding more (i.e. over-provisioning). This number ranges between 3 and 6 SPCSFs, even with the smallest dataset (\(T = 0.8\), with 6 SPCSFs). When there are few SPCSFs available, the IoU and satisfaction coverage tend to be smaller. This figure shows that between \(T = 0.0\) and \(T = 0.2\), where there is higher satisfaction and accuracy (Section 5.2.3), at least 6%-9% of the original or mildly filtered dataset \(DS_{8}\) is enough to deliver accurate results.

5.2.3 Accuracy↩︎

The accuracy is measured by (i) the satisfaction ratios, which measure how well each requirement to cover a security dimension is achieved, and is obtained through the ratio \(s_{r} =\) recommended / requested \(\mid s_{r} \in [0, 1]\); and (ii) the Normalised Discounted Cumulative Gain (NDCG) and (iii) the coverage ratios.

NDCG, in Eq. 20 , is a well-known ranking quality metric that measures how well an algorithm sorts items based on relevance. To do so, it compares the top \(K\) items (\(K\) being the cut-off point) of the ranking to the ideal order (based on relevance) and provides a ratio, where 1 means full alignment (higher quality or accuracy). It is important to point out that the relevance criteria (\(rel_{i}\)) per item \(i\) can differ depending on the purpose of the system. For instance, it can assume the ground truth as the set of items with highest total score, in descending order; or it can assume as ground truth those items that are the most precise in term of a particular use case.

\[\begin{gather} \label{eq:evaluation:internal:ndcg:base} \text{DCG}@K = \sum_{i=1}^{K} \frac{rel_{i}}{\log_{2}(i+1)} \\[0.2em] \text{NDCG}@K = \frac{\text{DCG}@K ~ \text{(obtained)}}{\text{DCG}@K ~ \text{(ideal)}} \end{gather}\tag{20}\]

In this case, both approaches were considered to generate the \(DCG@K\) ideal value (ground truth), using a subset of SPCSFs ordered by relevance: (i) of those with highest total score (\(rel_i^1 = f(\text{score\_total})\)); and (ii) of those that avoid under-provisioning and minimise over-provisioning, accommodating the most to the exact requirements (\(rel_i^2 = f(\text{coverage})\)). Both values are provided in Table 11 as "NDCG (Score)" and "NDCG (Min)", respectively. The first one, purely based on the "score_total" column from \(DS_{8}\) (where the top 20 high-scoring SPCSFs are shown in Table 2), yields 82%-99% on the clustered MAID and 52%-99% for the full MAID; both increasing along with \(T\) (i.e. with fewer SPCSFs to select from for the ranking). The second one, mostly focused on the "coverage_*" columns from \(DS_{8}\) (denoting contribution per SPCSF to an agent \(a\)), the values range between 99%-100% in the clustered MAID and 96%-100% for the full MAID. Given that both the utility function from the exploration stage, and the criteria of the recommender stage both point at ensuring there is no under-provisioning and avoiding over-provisioning, suggesting a minimal set of SPCSFs, it is natural that the second NDCG is much more aligned and stable. Another measurement comes from the coverage ratios. These are evaluated both using linear and geometric metrics and consider the security requirement array (\(\vec{\tau}\)), the security coverage array (\(a_{i}\)) and the number of agents (\(|A|\)). The best value these can achieve is 1, as it indicates full satisfaction/coverage.

On the one hand, the considered linear metrics are as follows:

• The satisfaction coverage, in Eq. 22 , measures the provided coverage per agent, ensuring its satisfaction to avoid under- or over provisioning.

• The resource efficiency, in Eq. 23 , measures over-
  provisioning, as an unnecessary bloat that adds nothing to cover the security requirements.

• The weighted Jaccard similarity index, or Intersection over Union (IoU), calculates how well the generated array (or shape) for the recommendation aligns with respect to the array (or shape) of the provided security requirements. This is computed in Eq. 24 and where \(x_{i} = a_{i}, y_{i} = {\tau_{i}}\) as a 1D array; yet the formula can also be used for a 2D plane – which is, after all, the graphical representation of the covered requirements.

\[\tag{21} \begin{align} \text{Satisfaction coverage} &= \frac{\sum_{a=1}^{|A|} \min(c_{a}, \tau_{a})}{\sum_{a=1}^{|A|} \tau_{a}} \tag{22} \\[1ex] \text{Resource efficiency} &= \frac{\sum_{a=1}^{|A|} \min(c_{a}, \tau_{a})}{\sum_{a=1}^{|A|} c_{a}} \tag{23} \\[1ex] \text{Jaccard (IoU)} &= \frac{\sum_{a=1}^{|A|} \min(c_{a}, \tau_{a})}{\sum_{a=1}^{|A|} \max(c_{a}, \tau_{a})} \tag{24} \end{align}\]

The average resource efficiency (Fig. 15) stays relatively constant, between 72% and 82%. This hints that the over-provisioning varies between 18% and 28%.

When the recommended SPCSFs are more specialised and better fit few expected security requirements, less over-provisioning is expected. The high variability observed as \(T\) increases could be derived from less specialised controls that may be providing extra, unrequested coverage. An example of this is visible in the single runs shown in Fig. 12 and Fig. 22; where some SPCSFs over-contribute and negatively impact this metric.

The satisfaction coverage, shown in Fig. 16, proves that until \(T = 0.3\), all security requirements are almost fully covered at 99%-100%. Beyond it, and due to the smaller datasets and lesser availability of SPCSFs, this coverage drops to 73% on \(T = 0.5\) (31 SPCSFs) and to \(\sim\)​40% from \(T = 0.7\) (6-13 SPCSFs).

Another way to look at how well the coverage is satisfied is the satisfaction ratio, which simply divides the recommended coverage between the requested one, per agent \(a\). See Fig. 17, showing the progression of the average satisfaction per \(a\) across each threshold \(T\). This plot provides detail on which \(a\) are most affected by the reduction of the dataset size as \(T\) increases: except for confidentiality, integrity and accountability, other SPCSFs are gradually affected or even drastically - the more with under-covered security dimensions for SPCSFs, as for non-repudiation and privacy (Table 3).

Fig. 18 depicts the IoU, diminishing steadily from the full dataset with 77% (\(T = 0.0\), with 97 SPCSFs) towards the minimal dataset (\(T = 0.8\), with 6 SPCSFs), with 35%. At \(T = 0.5\), the intersection of the recommended VS requested security requirements drops below 60%, impacted already by the lack of SPCSFs in the dataset with enough total score and dimension coverage.

On the other hand, the considered geometric metrics are:

• The area coverage, in Eq. 26 , assesses the percentage of the area of the requested security requirements that is also covered by the area of the recommended security controls. This indirectly measures under-provisioning; which is one of the key (utility) terms to minimise.

• The area efficiency, in Eq. 27 , indirectly measures over-provisioning; as it identifies which area from the recommended security controls contributes to fill/intersect the area from the requested security requirements.

\[\tag{25} \begin{align} \text{Area coverage} &= \frac{\text{Area}(\min(\vec{c}, \vec{\tau}))}{\text{Area}(\vec{\tau})} \tag{26} \\[1ex] \text{Area efficiency} &= \frac{\text{Area}(\min(\vec{c}, \vec{\tau}))}{\text{Area}(\vec{c})} \tag{27} \end{align}\]

Since this is a 2D plane, the area of each polygon with \(|A|\) and formed by a vector \(\vec{x}\) is computed for both metrics with Eq. 28 . Other common operations are the intersection vector between each element of the security coverage array (\(a_{i}\)) and the target array \(\vec{\tau}\), with \(\min(\vec{a}, \vec{\tau})\).

\[\text{Area}(\vec{x}) = \frac{1}{2} \sin\left(\frac{2\pi}{|A|}\right) \sum_{i=1}^{|A|} x_i \cdot x_{i+1} \mid x_{|A|+1} = x_1 \label{eq:evaluation:radar95area}\tag{28}\]

The area coverage is shown in Fig. 19. It follows a trend that is relatively similar to that in the satisfaction coverage (Fig. 16), demonstrating how between \(T = 0.0\) and \(T = 0.3\) the security recommendations are almost perfectly aligned with the security requirements and how it drops around \(\sim\)​50% accuracy starting in \(T = 0.5\), where only 32% of the initial implementable SPCSFs are then available.

Fig. 20 plots the area efficiency as highly variable (highest SD in Table 11). This lesser availability of SPCSFs for suggestion is likely introducing extra, not needed coverage on each of the requested agents.

5.2.4 Equity↩︎

The fairness or equity of the system is assessed by the Jain’s index in Eq. 29 . This index is commonly used to evaluate resource consumption in networks (e.g. assigned network flows), and provides a ratio between the sum of the resources to share (\(x_{i}\)), powered to two, and the number of users (\(n\)) and the quadratic sum of the same resources. In this context, \(n = |A|\) and resources are satisfaction ratios.

\[\label{eq:evaluation:jain-index} \mathcal{J}(x_{1}, ..., x_{n}) = \frac{\left(\sum_{i=1}^{n} x_i\right)^2}{n \cdot \sum_{i=1}^{n} x_i^2}\tag{29}\]

The outcome denotes the equity of the system in that the provided recommendations do not sacrifice too much to cover a given agent at the expense of others. This situation is more evident upon negative synergies across clusters or SPCSFs. The perfect ratio, and maximum boundary, is 1; with the minimum being \(1 / n\).

In this case, the Jain’s fairness (Fig. 21) is stable at 1 until \(T = 0.3\), after which it stabilises at 0.81-0.82 until \(T = 0.6\) and degrades until reaching 0.49.

5.3 Use case evaluation↩︎

The 7 considered security dimensions are already considered as part of the 8 security goals, extended from the CIA triad in the Reference Model of Information Assurance & Security (RMIAS) or IAS-Octave [8] framework. The authors provide a qualitative analysis of an organisation’s information security policies, qualitatively mapping a subset of these to the CIA triad and the accountability security dimensions. Similarly, authors from [56] analyse a Bring Your Own Device (BYOD) environment and suggest that both the CIA, accountability and auditability (potentially considered as a group under authenticity), privacy and non-repudiation security dimensions are required to ensure traceability of users’ devices for post-incident analyses.

Based on the above, this section first performs a high-level analysis of an IT enterprise environment to extract its requirements in terms of security dimensions, roughly quantifying these. These are used as a specific use case to validate against the proposed recommender. A common, non-critical IT enterprise delivers solutions that are often protected by Intellectual Property Rights (IPR), leverages private/public cloud resources to carry out the work and may allow BYOD to some extent. The security of the data generated is subject to IPR and guided through institutional security policies. Likewise, the proper functioning and security of the hardware and tools in use is dictated by security policies that monitor and alert about malfunctioning, filter specific applications or actions within a system and continuously monitor the security posture of both internal and BYOD devices to grant (Zero Trust) access to network segments and the resources accessible by these. In this sense, such an environment emphasises data confidentiality to conform to the IPR internal regulation. It also requires maintaining good availability and integrity for both internal and potentially customer-exposed data and services, as well as a degree of privacy when processing human data to cover regulatory requirements. Authenticity, accountability and non-repudiation are needed to minimise unauthorised access to such resources and to identify who did what when analysing incidents. Considering this, an operator setting up the supporting environment for such organisation may quantify security requirements \(\vec{\tau}\) as those from Table 8.

The recommender is run once with the indicated requirements, testing both modes \(M\) and methods \(\mathcal{L}\) and using both the full pool of SPCSFs (\(T=0.0\)) and a constrained one (\(T=0.4\)) to assess the changing behaviour. Fig. 22 and Table 8 represent the DSS outcomes in graphical and tabular form: the suggested SPCSFs and the coverage \(\vec{c}\) per security dimension. This is shown here across modes \(M\) and \(T\) values whilst pinning \(\mathcal{L}\) to EXP3 for brevity. Among the proposed SPCSFs are those preserving integrity (SI-3: Malicious Code Protection, AU-9: Protection of Audit Information), availability (AC-7: Unsuccessful Logon Attempts) or authenticity (IA-4: Identifier Management). Table [tbl:tab:evaluation:use-case] shows averaged values for relevant metrics across both learning methods \(\mathcal{L}\).

This use case is specially illustrative as it exhibits both over- and under- provisioning — where the latter is even more critical to avoid, as indicated in Section 4.3.3. Over-provisioning occurs at \(T=0.0\), with all available SW-implementable SPCSFs (\(N=97\)): Fig. [fig:evaluation:use-case:clust-exp3-t00] and Fig. [fig:evaluation:use-case:full-exp3-t00] show a spike at Nr and offering maximum coverage, well beyond the explored decision space and the targeted security requirements that directly or indirectly act as bounds. Only 2 SPCSFs cover Nr (Table 3), namely AU-8 and AU-10; which offer coverage of 0.2 and 1 and have a total score of 0.071 and 0.392, respectively. AU-10 can thus avoid under-provisioning. Under-provisioning happens at \(T=0.4\), with \(N=37\) SPCSFs: there, Fig. [fig:evaluation:use-case:clust-exp3-t04] and Fig. [fig:evaluation:use-case:full-exp3-t04] show a lack of coverage on Nr because AU-10’s total score is just below the filtered \(T\). Pr also suffers from under-provision at this point, since the only 2 SPCSFs after the score filtering are AC-19 and SC-16 with coverage of 0.1 and 0.2, respectively. With such a limited pool of SPCSFs, these cannot be fully met.

5.4 Evaluation against baseline↩︎

A greedy recommender that is bounded and filtered was implemented as baseline for the evaluation. Similar to the dynamic pruning stage at Algorithm 11 (lines [alg:implementation-model:gt-exploration-recommendation:line:s15]-[alg:implementation-model:gt-exploration-recommendation:line:s23]), it also applies a budget \(B\) and upper bounds \(\vec{\beta}\) (the direct security requirements \(\vec{\tau}\) ) to limit over-provisioning. This baseline first collects the coverage and score per SPCSF from the dataset, then sorts these using that score and iterates through these while the requirements are still not met and the budget is not exhausted. Only SPCSFs that increase the existing coverage while not exceeding the requirement bounds are added. Ultimately, it performs another pass to filter out redundant controls and thus suggest a minimal subset.

Table 10 roughly compares two greedy recommenders with (i) a naive one and (ii) a bounded, filtered one used as baseline; and (iii) the recommender proposed in this work. It shows the aggregated average and SD values resulting from 100 iterations and across all full and clustered modes, displaying a subset of relevant metrics and thresholds, where \(T=0.3\) is the turning point for all recommending approaches (Section 5.5). The proposed recommender shows improved values for all metrics, except for the recommendation time \(\mathcal{R}\), since it has to run extra steps on e.g. the received profile \(\sigma^{*}\) and the mode \(M\) processing (clustered vs full). It is also worth noting that, whilst directly using \(\vec{\tau}\) as \(\vec{\beta}\) is enough to maximise coverage, it does actively ignore the identified synergies across security dimensions and the overall agreement, calculated in the utility functions (Section 4.3) to also minimise over-provisioning — which is not sufficiently addressed by the filtering logic during the recommendation stage.

Table 12 compiles the detailed baseline values and deviation for each metric and threshold, and it is compared against the corresponding values for Hedge and EXP3 in the rows named \(\Delta_{1}\) and \(\Delta_{2}\), respectively. Improving \(\uparrow\) and worsening \(\downarrow\) indicators are placed next to each value. The time (performance, \(\mathcal{R}\)) taken by both approaches is in the order of milliseconds, where the baseline halves that of the proposed recommender. This is so because the latter needs multiple passes to fine-tune SPCSF selection and limit SPCSF over-provisioning when issuing recommendations, whilst maintaining a high accuracy. Such accuracy is consistently higher for all dataset sizes (\(T\)) of the proposed recommender compared to that of the baseline and is as a trade-off for the slight increase in time.

5.5 Analysis↩︎

Regarding the internal evaluation (Section 5.2), the satisfaction ratio, area coverage and fairness are of 97%-100% between \(T = 0.0\) and \(T = 0.2\) (i.e. with 68-97 SPCSFs). There, 6 SPCSFs are needed to achieve a high accuracy. Between \(T = 0.3\) and \(T = 0.4\) (37-63 SPCSFs), there is already a significant drop in the satisfaction and area coverage along with the IoU. However, the satisfaction coverage of the security requirements is still good for \(T = 0.3\) (99%). \(T = 0.3\) acts as the turning point where many metrics degrade for both the proposed recommender and the baseline. Starting from \(T = 0.5\), the dataset only provides 31 SPCSFs; with the satisfaction coverage dropping below 80% and the IoU dropping below 60%, denoting severe under-provisioning; which must be avoided.

Whilst there are some side benefits to a small number of SPCSFs (higher resource efficiency with less over-provisioning, a PoA that gradually reduces to its theoretical target and a steadily increasing NDCG), such a small dataset is clearly not enough to fulfil the requirements. Summing up, the dataset \(DS_{8}\) should not be heavily filtered and rather rely on adequate (likely customised) budget controls that can limit the recommendation of SPCSFs to ensure the operator does not incur too much effort to achieve the desired degree of security. With at least 63 SPCSFs, the satisfaction coverage, satisfaction ratio and NDCG stay close or equal to 100%.

When compared against the baseline (Section 5.4), this work shows a consistent improvement in area efficiency, resource efficiency and profile IoU (up to \(21.43\%\), \(10.59\%\) and \(10.70\%\) for \(T=0.0\)), with improvements up to \(T \le 0.3\). The provisioning-conscious ranking NDCG score, which recommends minimal SPCSF sets, is naturally higher for all \(T\) (up to \(27.09\%\) in \(T = 0.6\)). Also, the cost (numbers of controls used) is slightly smaller until \(T \le 0.3\), reducing up to \(6.03\%\) for \(T = 0.0\). This suggests some better adjustment and minimal over-provisioning than that from the baseline.

6 Discussion and conclusion↩︎

This section discusses the overall idea and behaviour of the proposed dataset and DSS, considering their assumptions and limitations; and concludes by summarising the contributions and future lines of work to consider.

6.1 Discussion↩︎

The proposed DSS is targeted at system operators with limited or no cybersecurity background, aiming to guide them on the initial security setup. It thus assumes that cybersecurity experts are not in the loop to further curate or validate the dataset – although it relies on well-known frameworks and adaptable data processing to incorporate new partial scores and recalculate the total one ultimately used by the recommender.

The dataset is constructed by correlating, transforming (i.e. often normalising) and aggregating multiple sources, keeping its average as partial and total scores. The latter is used as a normal distribution in the MAID chance nodes to account for potential minor variability in the scores from the list of SPCSFs (Section 4.4.1).

From the evaluations in Section 5 it can be extracted that DSS performs best when it can select at least from \(N = 63\) SPCSFs (\(T = 0.0\) to \(T = 0.3\)), achieving an accuracy (measured as area coverage and satisfaction ratio) of 97%-99% and 99%-100%, respectively. \(T = 0.3\) is seen as the turning point, where these drop to 60% and 80% but the system works well. After this, a massive drop in area coverage suggests under-provisioning problems, which should be avoided as it noticeably degrades the recommendation due to the narrower coverage of the remaining SPCSFs towards the security dimensions. For best results, an organisation should cover at least 63 SPCSFs; although with at least 28 SPCSFs it could still deliver reasonably well for well-covered security dimensions (e.g. others than accountability and privacy, as per Section 5.3).

6.2 Limitations↩︎

This section documents how some assumptions from the dataset generation and the recommendation algorithm introduce limitations that must be considered.

6.2.1 Data-related limitations↩︎

The dataset is obtained by correlating existing peer-reviewed sources of different types and sources (Table 2). When correlating data of different granularity and size, such aggregation can incur loss of detail. This happens for \(DS_{2*}\) [14], which maps expert-provided scores to 36 security processes in their taxonomy and to other frameworks (e.g. NIST CSF). The translation between NIST CSF to NIST SP 800-53 rev5 relates 20 categories from the former to 300 SPCSFs from the latter; which introduces repeated scores for different types of SPCSFs and reduces its variability and discriminative potential. Another aspect to consider is that the dataset is explicitly biased towards SW-implementable SPCFs, which inherit higher scores from the source datasets and result in a higher total score which keeps these available across higher \(T\) during the dataset filtering.

Interpreting large amounts of descriptions for the SPCSFs from the OSCAL catalogue [13] and online resources [28] was delegated to a GenAI model (Gemini 2.5 Pro) to create \(DS_{4}\) and identify how much a SPCSF contributes to preserve a security dimension. The reader may assess the validity of this mapping by reviewing \(DS_{8}\) the "coverage_justification" column, explaining the rationale behind the assignment. An alternative is to perform a deterministic, replicable text parsing and analysis, with coverage being inferred through the frequency of keywords and their weight towards security dimensions based on authoritative sources like OSCAL [13], CNSSI 1253 [30] or community-backed resources [57]. However, it is worth noting that human subjectivity can be brought into the process directly (e.g. expert survey, as in \(DS_{2*}\)) or indirectly (through GenAI or assumed keywords and weights for text analysis). A highly robust approach would possibly involve formal modelling of security dimensions and protected resources, running sandboxed emulated or real attacks and mitigating mechanisms and observing the impact of both.

Besides, the dataset provides only up to the granularity of SPCSFs. It would be, however, interesting to also relate SPCSFs to categories of mechanisms, and mechanisms to specific frameworks or tools. [58] shows an example with custom security controls, categories and mechanisms.

6.2.2 Recommender-related limitations↩︎

The recommender process also runs on some assumptions that impact its outcome. The DSS algorithm discretises the decision space into four categories (Section 4.4.2) that align with traditional security baselines used in InfoSec and regulatory sources. This decreases the exploration space for a game-theoretic approach at the cost of less granularity in this stage, thus requiring a fine-tuning (recommending) stage to provide fine-grained coverage and identify attribution of SPCSFs to security dimensions. Combining (i) fine-grained profiles (e.g. based in coverage deciles) and (ii) a suitable application of the Shapley value (assuming a coalition-based game) in the utility functions to extract attribution could possibly remove the second recommending and pruning stage. Regarding its output, it does not return a pure deterministic outcome across varying \(M\) values (although it seems to do so for \(\mathcal{L}\)), as sometimes different SPCSFs can be suggested depending on the number of clusters and their arrangements. See Table 8 and Fig 22. On the other hand, coverage and scoring are here simplified to few attributes; yet a more advanced model should consider the nuances of the resources to protect and adequately expand the representing features or dimensions.

6.3 Conclusion and future work↩︎

This work presented a scalable, accurate and fair security DSS to recommend NIST SP 800-53 rev5 security and privacy controls (SPCSFs), leveraging for decision a dataset extended from academic and InfoSec sources; which can be further adapted by users with cybersecurity exposure and extended by researchers. It is implemented as a game-theoretic MAID, where multiple agents -representing the security dimensions to cover- apply no-regret learning to each utility function, maximising its payoff. The outcome is the recommended subset of SPCSFs that best meet the provided security requirements from the available SPCSF pool.

Future steps to evaluate in this research currently consider three fronts: (i) easing the extraction of the user-defined coverage for the security requirements from other kinds of datasets, such as those containing a subset of relevant threats for an organisation; (ii) evaluating games modelling the adversarial behaviour based on its profile (i.e. applying specific tactics and techniques), potentially integrating the results of real or simulated campaigns as feedback to the modelling of the behaviour for the defenders; as well as (iii) improving the granularity and completeness of recommendations through an open source-derived taxonomy relating SW-implementable mechanisms for defenders related to both NIST SP 800-53 rev5 and MITRE ATT&CK.

Generative AI tools usage declaration↩︎

During the preparation of this work, the authors used Gemini 2.5/3 Pro, GPT 5.2 and Google NotebookLM for academic literature exploration, InfoSec interpretation, code prototyping, LaTeX editing, figure rendering and language revision. GenAI tools were used in a limited and supervised manner. All results were reviewed, corrected, and validated by the authors. The authors take full responsibility for the content of this article.

CRediT authorship contribution statement↩︎

Carolina Fernández-Martínez: writing – review & editing, writing – original draft, visualisation, validation, software, resources, project administration, methodology, investigation, formal analysis, data curation, conceptualisation. Shuaib Siddiqui: writing – review & editing, project administration, funding acquisition, supervision, conceptualisation. Vanesa Daza: writing – review & editing, supervision, conceptualisation.

Code and data availability↩︎

The dataset generated in Section 3 and the code used to generate it are available at [59]. Both the dataset and the code used in Sections 4 and 5 are available at [60].

Declaration of competing interest↩︎

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Funding↩︎

This work was supported by the grant COALESCE-6G PID2024-163028OB-I00, funded by MICIU/AEI/10.13039/501100011033/FEDER, EU.

Acknowledgements↩︎

The authors would like to thank Estela Carmona-Cejudo for her advice on improving the mathematical formulation and the consistency and correctness of the utility function; as well as Amir Ansari for his insights on the MAID modelling and working sessions on BN and probabilistic-related validation. The authors also acknowledge the Spanish Recovery, Transformation and Resilience Plan through the European Union (Next Generation) under grant AEI-PID2021-128521OB-I00.

7 ↩︎

Table 11 details the full evaluation of the recommender, whereas Table 12 shows the comparison against the greedy, filtered baseline. These can be found as appendices.

3pt

References↩︎

---

1. © 2026. This manuscript version is made available under the CC-BY-NC-SA 4.0 license. The final published version of record is available in Knowledge-Based Systems at https://doi.org/10.1016/j.knosys.2026.116558.
   Corresponding author↩︎