AutoSUT: The Environment Semantics Gap in Structured CTI
for Adversary Emulation
January 01, 1970
Structured Cyber Threat Intelligence (CTI) is increasingly used to support adversary emulation, detection evaluation, and cyber range design. However, each of those workflows still needs a target System Under Test (SUT) whose environment is not fully described by the public CTI. We measure how much of that environment is available in MITRE ATT&CK Structured Threat Information Expression (STIX) bundles. Across the ATT&CK Enterprise, Mobile, and Industrial Control Systems datasets, with the Common Attack Pattern Enumeration and Classification (CAPEC) and 5G Hierarchy of Threats (FiGHT) as contrast datasets, we report platform coverage, software specificity, vulnerability evidence, and deployment compatibility. Platform tags are common, but software references rarely pin versions or Common Platform Enumeration (CPE) identifiers. In Enterprise, 97.6% of software objects lack both, and campaign-level Common Vulnerabilities and Exposures (CVE) evidence remains sparse and fragmented.
The measurements show that public ATT&CK-style structured CTI narrows candidate environments and supports lower-bound backend-family assignment, but structured fields alone do not justify deriving a replay-ready SUT. Profile confusion falls from 1.3% at one linked software item to 0% at two. These numbers locate the boundary between corpus-supported environment narrowing and the version, vulnerability, and deployment detail that must still come from outside the corpus. Beyond locating the boundary, we are the first to show it has an operational consequence: holding every corpus-supported element fixed and varying only the analyst-authored region yields multiple distinct, campaign-compatible SUTs, including an executable witness that runs the same real vulnerability. Structured CTI, therefore, constrains but does not uniquely determine the environment. Replay-ready emulation should accordingly declare which environment commitments the corpus supports and which remain analyst-authored.
cyber threat intelligence, STIX, MITRE ATT&CK, system under test, adversary emulation, structured CTI
Structured Cyber Threat Intelligence (CTI) is increasingly used to support adversary emulation, detection evaluation, cyber range construction, and attribution analysis. The execution-facing workflows among them need two inputs: a description of what an adversary does and an environment in which those actions can run. The first is increasingly well served by the MITRE ATT&CK corpus, which catalogs 835 Enterprise techniques across 187 documented groups and 52 campaigns, all encoded in machine-readable Structured Threat Information Expression (STIX) bundles [1], [2]. The second remains structurally underspecified. Each replay or evaluation use still needs a System Under Test (SUT): a target environment with the intended operating system, software stack, exposure surface, and vulnerability state. If that environment is wrong, even a correct procedure cannot produce a reproducible emulation result.
We therefore define the environment semantics gap 1 as a measurable property of structured CTI: the SUT information required to instantiate a replay-ready environment that cannot be derived from structured CTI alone. We ask: what is the strongest lower-bound SUT claim that public ATT&CK-style structured CTI can justify without external enrichment? The input is public structured CTI in ATT&CK-style STIX bundles. The output is a bounded SUT claim: which environment family the corpus narrows, which backend family it justifies, and which layers remain analyst-authored. AutoSUT is the measurement pipeline: it locates where corpus-supported narrowing ends and analyst specification begins, without substituting exploit synthesis, command generation, or runtime inference for missing CTI. We then add one controlled construction, a non-uniqueness demonstration (Section 7), that varies only the analyst-authored region to show the measured gap has an executable consequence.
Across the ATT&CK Enterprise, Mobile, and Industrial Control Systems (ICS) datasets, with the Common Attack Pattern Enumeration and Classification (CAPEC) and the 5G Hierarchy of Threats (FiGHT) used as coverage contrasts, the pattern is consistent. Platform evidence is common but coarse. Software references rarely pin versions or Common Platform Enumeration (CPE) identifiers. Common Vulnerabilities and Exposures (CVE) evidence is sparse and often split between structured fields and free text. The result is a reproducible quantitative boundary between what public datasets encode and what analysts must still supply.
That boundary matters to every downstream consumer of structured CTI. Red-team platforms execute only after the environment has been declared [3]–[7]; MITRE Engenuity ATT&CK evaluation programs still depend on manually declared representative environments [8]; provenance-graph detection systems build labs whose representativeness depends on how completely CTI specifies the target environment [9]–[12]; and attribution confidence is bounded by evidence quality rather than by the absence of behavioral hypotheses [13], [14]. An under-specified environment context can therefore distort detection evaluation, emulation fidelity, and attribution claims alike [15]. Many execution-facing workflows implicitly rely on structured CTI to narrow the environment in which documented behavior runs; how much of that environment the corpus actually determines is not directly measured.
ATT&CK is an appropriate public corpus in which to test this boundary because it is the most widely reused structured CTI source for threat-informed defense and because its public STIX bundles expose behavior-centric evidence in a machine-readable form [1], [2], [8], [16]. The question is not whether ATT&CK or STIX should be a lab generator, but how far their public structured fields already determine the environment before analyst authorship resumes. We therefore isolate the environment dimension from the procedural one. This leaves four research questions (RQs):
RQ1: How much structured CTI encodes the platform, software, and vulnerability details needed for SUT construction?
RQ2: How far can those signals support lower-bound SUT and backend-family claims, and where do they stop?
RQ3: How specific are SUT requirement profiles for campaigns and intrusion sets without over-claiming attribution?
RQ4: Given the corpus-supported elements a SUT must preserve, does structured CTI determine a unique environment, or admit multiple compatible ones?
This paper provides the first measurement of environment derivability from ATT&CK-style structured CTI, namely how much of a replay-ready SUT the public corpus determines, as distinct from the behavioral and field-completeness quality that prior studies assess. The contributions are fourfold. We define a SUT-centric measurement model for the environment semantics gap; measure where platform, software, vulnerability, and backend signals remain strong enough for coarse narrowing and where they stop; quantify bounded enrichment, including a narrow CVE-to-package concretization slice; and constructively demonstrate environment non-uniqueness with an executable witness for CVE-2021-41773 (Section 7). Taken together, these results support a single thesis: structured CTI fixes an invariant environmental core but leaves a free region, so it constrains but does not uniquely determine the executable environment, and that under-determination has a demonstrable, executable consequence.
The MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) framework documents adversary behavior as tactics, techniques, and sub-techniques across three domains: Enterprise, Mobile, and ICS. Enterprise is the largest of the three, containing 835 techniques and sub-techniques across 14 tactics in v18.1 [1], [2]. ATT&CK content is distributed as STIX 2.x bundles [17]. For SUT construction, the most relevant object types are attack patterns, campaigns, intrusion sets, malware, tools, and their relationships (Table 1). In the public ATT&CK bundles we measure, STIX captures semantic associations such as uses, attributed-to, and targets; these objects do not, by themselves, specify deployment-ready environment constraints at the granularity needed for SUT derivation.
| Entity | SUT relevance |
|---|---|
| attack-pattern | Platform hints via x_mitre_platforms; system requirements in x_mitre_system_requirements (rarely populated). |
| malware / tool | Implies OS dependency; version rarely specified; CPE mappings largely absent. |
| campaign | Scope operation to the time window and target set; links to techniques and intrusion set. |
| intrusion-set | Aggregates behavioral profile; useful for SUT susceptibility profiling but not for single-campaign instantiation. |
| vulnerability | CVE-mapped objects are present in some bundles; they are rarely linked directly to campaigns. |
Red-team platforms such as Caldera execute adversary procedures against a declared target environment rather than deriving that environment directly from CTI [3]. SUT construction, therefore, remains a manual precondition for emulation, which is exactly the burden this paper measures.
Here, environment denotes the complete execution context required to reproduce the behavior described by a CTI object. This includes the operating system family and version, installed software (with versions and configurations), network topology and reachability, identity and privilege model, and any exploitable vulnerabilities.
An environment description is machine-actionable for SUT derivation when every constituent constraint is expressed in a structured, parseable field such as x_mitre_platforms, a CPE string, or a CVE identifier, rather than in
free-text prose. The System Under Test (SUT) is the concrete instantiation of such an environment in a virtualized or containerized lab.
Each CTI object can contribute three kinds of SUT constraints. Platform constraints are defined by x_mitre_platforms and indicate an OS family, cloud provider, or device type. Software constraints arise when a malware or
tool object is linked via uses relationships to a technique or campaign, implying that some installable software component must exist in the SUT. Vulnerability constraints arise when a CVE identifier appears in descriptions or
when a vulnerability object is linked to a campaign, technique, or software object.
For each campaign, and analogously for each intrusion set, we define a SUT requirement profile as the union of those three components. The profile is an analysis construct rather than an execution plan. It is static with respect to a fixed bundle snapshot: given one corpus release, the pipeline emits the strongest lower-bound SUT claim justified by that snapshot. It does not adapt online during execution. Declared step-conditioned overlays can still be applied at execution time, such as exposing a service boundary or staging a weak credential immediately before one technique, but those overlays are execution support authored outside the corpus rather than new inference from CTI. For reporting, we distinguish coarse profiles (software plus platform), anchored profiles (coarse plus version/CPE or campaign-level CVE), and exploit-pinned profiles (coarse plus campaign-level CVE).
Viewed operationally, the measurement problem has three layers. The observed evidence space is the set of structured platforms, software, and vulnerability signals present in one fixed bundle snapshot. The derivable claim space is the strongest lower-bound SUT and backend-family claim justified by those signals under fixed rules. The analyst-authored remainder is the version, configuration, topology, trust-boundary, and exposure detail still required for replay-ready instantiation. This paper measures the boundary between the derivable claim space and that remainder.
Separately, we classify each technique by the minimum backend needed to satisfy its constraints: container-feasible for user-space behaviors without kernel, firmware, or Active Directory dependence; virtual-machine-required for kernel interaction, process injection, or boot-level persistence; and infrastructure-dependent for behaviors that presuppose multi-host topology, domain services, or enterprise identity.
Each layer only narrows the candidate SUT family or the minimum backend family (Figure [fig:measurement95model]); it never fills the missing remainder. Coarse profiles require platform-plus-software evidence. Anchored profiles add version/CPE or CVE evidence. Exploit-pinned profiles add campaign-level CVE evidence to a coarse profile, thereby identifying a plausible vulnerable deployment family even when the exact lab topology remains analyst-supplied.
Following the measurement model in Section 3, we select datasets that expose the same core STIX structures while covering different ATT&CK scopes. The MITRE ATT&CK Enterprise bundle v18.1 is the primary source for all quantitative analyses because it is the largest domain by object count and carries the campaign coverage that RQ2 and RQ3 require. Table 2 reports the raw object counts across the five sources that this study traverses.
4pt
| Source | Techniques | Campaigns | IS | Malware | Tools |
|---|---|---|---|---|---|
| Enterprise | 835 | 52 | 187 | 696 | 91 |
| Mobile | 190 | 3 | 17 | 121 | 2 |
| ICS | 95 | 8 | 16 | 30 | 0 |
| CAPEC | 615 | 0 | 0 | 0 | 0 |
| FiGHT | 707 | 0 | 136 | 475 | 73 |
RQ2 and RQ3 remain focused on Enterprise because only Enterprise contains sufficient campaign coverage for deployment-oriented analysis; CAPEC is included as STIX-encoded attack-pattern knowledge for contrast in coverage. All RQ metrics use active-object counts after deprecation and revocation filtering. In Enterprise, this filter leaves 691 active techniques, 52 campaigns, 172 intrusion sets, and 784 software objects (693 malware \(+\) 91 tools), reachable through 20048 relationship objects.
The pipeline applies explicit extraction and decision rules. We first parse and normalize the bundles, then define the compatibility taxonomy used for deployability claims, bound the enrichment steps allowed beyond structured fields, and specify the profile-specificity metric used in RQ3. Each reported number, therefore, maps to a stable rule rather than to ad hoc interpretation.
Evidence extraction. We parse five locally available STIX bundles as JavaScript Object Notation (JSON) and normalize object identifiers. Enterprise, Mobile, and ICS drive the core analyses; CAPEC and FiGHT support cross-corpus coverage in RQ1.
We exclude deprecated and revoked objects, which removes 144 of the 835 Enterprise attack-pattern objects, leaving 691 active techniques. Relationships are indexed by (source_ref, relationship_type, target_ref) triples to enable
bidirectional traversal. For each campaign, the set of techniques is collected from all attack-pattern objects reachable via uses relationships; software and CVE associations are derived analogously.
From attack-pattern objects, we extract platform scope, host requirements, privilege requirements, and defense-bypass cues. From malware and tool objects, we extract names, aliases, platform tags, and any CPE or version strings in
external_references. CVE identifiers are extracted from description fields and from linked vulnerability objects.
For campaign-level target-platform inference, we adopt a strict software-only rule. Platforms are inferred only from software-linked platform tags. Technique-level platforms are excluded from this step because they encode general compatibility rather than campaign-specific target evidence. Campaigns with no linked software are reported as platform unknown rather than receiving inferred values. This avoids turning generic compatibility into campaign-specific target claims.
Compatibility taxonomy. The taxonomy is an operational lower-bound abstraction for backend-family assignment, not a claim about full execution fidelity. Each technique receives exactly one compatibility class under a fixed precedence order: container-feasible (CF), virtual-machine-required (VMR), or infrastructure-dependent (ID). Identity and infrastructure cues dominate, host-semantic VM cues come next, and narrow container-feasible cues come last. Any remaining unresolved technique defaults to VMR.
More concretely, we assign ID when platform tags reference Windows Domain or Identity Provider, or when a lateral-movement technique depends on Active Directory software. We assign VMR when
permissions-required fields indicate administrator privileges (Administrator, SYSTEM, or root), and likewise for privilege-escalation and boot/firmware defense-evasion techniques. We assign CF when x_mitre_platforms is
restricted to Containers or Linux and the technique is not in kernel-interaction or boot-persistence clusters.
This taxonomy is a conservative proxy for deployability rather than an execution ground truth. Explicit rules cover 32.7% of techniques; the remaining 67.3% default to VMR so that unresolved cases overestimate rather than underestimate infrastructure cost. CF is narrow, ID captures explicit enterprise-service dependence, and the unresolved middle goes to VMR because the safer failure mode for this paper is false VMR inflation rather than false CF optimism.
3pt
| Class | Structured trigger | Operational reading and preferred conservative error |
|---|---|---|
| CF | Platforms restricted to Linux/Containers; no kernel, boot, or identity cues | Candidate for container-only replay. The preferred error is false exclusion from CF rather than false CF optimism. |
| VMR | Privilege, kernel, firmware, or boot semantics; also, the unresolved residue | Requires host-level VM semantics. The preferred error is false VMR inflation when structured fields are too weak to justify CF or ID. |
| ID | Domain, identity-provider, or Active Directory semantics; lateral movement with enterprise-service dependence | Requires enterprise services or multi-host structure. The class bounds backend family, not the exact topology, trust boundary, or host count. |
Under that conservative error budget (Table 3), if all unresolved techniques were optimistically forced to CF, the non-CF share would still be 29.9%; under VMR or ID defaults, it rises to 97.2%. More importantly, the explicit-rule subset is already 91.6% non-CF. The fallback policy, therefore, changes the absolute split, but not the substantive conclusion that container-feasible techniques are a small minority and that unresolved cases mostly inflate VMR rather than hide CF coverage.
For independent auditing, we draw a stratified manual-validation sample of 36 techniques, balanced across CF/VMR/ID predictions and covering both explicit rules and fallback-driven VMR cases, so that the narrow CF minority, the explicit ID evidence, and the conservative middle can be inspected separately. The RQ2 percentages remain deterministic rule outputs, while this sample provides the exact rows needed to adjudicate the construct independently.
Bounded enrichment. To quantify the practical impact of the measured gaps, we evaluate three bounded enrichment steps over the ATT&CK bundles used in this study: description-based regex mining for software versions, free-text extraction of campaign-level CVE references, and deterministic compatibility assignment with conservative fallback for unresolved techniques. By headroom we mean the additional fraction of environment signal that becomes machine-actionable after one such enrichment step is applied to the current corpus. Each step yields auditable intermediate results. The resulting metrics quantify that recoverable margin without claiming full execution fidelity.
Profile specificity. For each intrusion set \(g\), we encode its SUT profile as a binary vector over all software and CVE identifiers in the corpus. Software membership is keyed by STIX object reference rather than by name, so alias lists carried on a malware or tool object cannot split that object in the profile. The metric is reproducible over the published bundle, but it is not a product-family canonicalization across distinct STIX software objects. Pairwise Jaccard distances are computed, and the fraction of groups with at least one neighbor within \(\delta = 0.1\) is reported as the SUT profile confusion rate. This metric uses unweighted set overlap because the profiles themselves are sparse sets rather than weighted timelines; Jaccard distance is \(0\) for identical support and \(1\) for disjoint support.
We use \(\delta = 0.1\) as a near-neighbor threshold, so two profiles count as confusable only when at least 90% of their union is shared. Operationally, this is a screening threshold: if two groups differ on fewer than 10% of the software/CVE support used to build their SUT profiles, a lab built for one should not be treated as actor-specific without additional evidence. Reporting the same rate across \(k \geq 1\), \(k \geq 2\), and \(k \geq 3\) linked-software thresholds separates genuine environmental overlap from ambiguity due to sparse evidence. Section 6 reports the measured values under these rules.
Throughout, the measured object is the public structured CTI surface and the claim-to-evidence boundary it supports; any execution substrate serves only as a compatibility check on that surface, never as the object of measurement.
We present the results as one evidence stack. First, we ask how far public datasets narrow the candidate environment. Next, we ask which replay substrate those signals justify once backend semantics are made explicit. We then test whether the resulting SUT profiles are sufficiently distinctive for evidence-filtered actor screening, rather than only for coarse lab selection, and finally quantify how much bounded enrichment is recovered without altering the underlying corpus. Unless otherwise stated, all results in this section should be read as lower-bound SUT and backend-family claims derived from structured fields plus the bounded enrichment steps defined in Section 5.
Evidence coverage. Of the 691 active Enterprise techniques, all 691 (100.0%) carry at least one entry in x_mitre_platforms, so platform annotation is effectively mandatory in current ATT&CK curation. In contrast,
x_mitre_system_requirements, the most specific platform field, is populated in 0 techniques (0.0%). Structured, fine-grained OS and software-version constraints are absent in the Enterprise environment.
Platform-field coverage is also high in Mobile (100.0%), ICS (98.8%), and FiGHT (100.0%) under the same active-object filter. CAPEC does not use x_mitre_platforms in this dataset because it encodes abstract attack patterns rather than
platform-bound behaviors.
Across datasets, Figure [fig:coverage95cross95corpus] shows the same pattern: platform-field coverage is high, software-link coverage is
moderate, and CVE-linked coverage is low. In CAPEC, the zeros are structural properties of the bundle: active attack-pattern objects link to no malware or tool objects and carry no CVE mentions. They are not missing
data caused by the measurement pipeline.
Applying the software-only platform rule to all 52 Enterprise campaigns yields a platform signal for 47 campaigns (90.4%); the remaining 5 campaigns (9.6%) are genuinely unknown under structured fields alone. These campaigns are FrostyGoop Incident, KV Botnet Activity, Leviathan Australian Intrusions, SPACEHOP Activity, and ShadowRay. Admitting technique-level platform tags would resolve all 5 unknowns, but at a cost that confirms why the rule excludes them: single-OS-family campaigns collapse from 22 to 1, while triple-OS cross-platform campaigns rise from 20 to 51 of 52. Technique tags remove uncertainty by making almost every campaign appear to target every operating system, which is a compatibility envelope rather than a target, so the software-only rule keeps the 9.6% unknown in exchange for an OS-narrowing signal that targeting claims can use.
Among campaigns with signal, the OS-family distribution (Table 4) is inferred from software-linked platform tags in ATT&CK rather than from explicit campaign target declarations; therefore, the values should be read as evidence of compatibility. Campaigns may be multi-OS, so OS-family counts can exceed the number of campaigns with signal. When the platform signal is driven by a single cross-platform tool (e.g., Linux/Windows/macOS tags), the inferred set is a compatibility envelope rather than a confirmed target-OS declaration.
| OS family | Campaigns | % signal | % total |
|---|---|---|---|
| Windows | 42 | 89.4% | 80.8% |
| Linux | 23 | 48.9% | 44.2% |
| macOS | 21 | 44.7% | 40.4% |
| Platform unknown | 5 | n/a | 9.6% |
How much that signal truly narrows the environment is mixed (Table 5). Of the 44 campaigns with any OS-family evidence, 22 collapse to a single family, 2 remain dual-family, and 20 still expand to a Linux, Windows, and macOS envelope. Within the 17 campaigns driven by exactly one linked software item, 10 narrow cleanly while 7 still expand to a triple-family compatibility envelope.
| Case | Campaigns |
|---|---|
| Single OS family | 22 |
| Two OS families | 2 |
| Linux, Windows, and macOS envelope | 20 |
| Single-OS, one-software case | 10 |
| Triple-OS, one-software case | 7 |
Another 3 campaigns carry only non-OS platform tags, such as network-device labels, and therefore do not narrow the OS family at all under the software-only rule. The platform signal is common, but it often remains broad.
OS-family evidence is, therefore, compatibility evidence for SUT construction, not a standalone attribution signal. The campaigns themselves are not behaviorally sparse: the 52 Enterprise campaigns use 19.6 techniques on average (median 17) across 8.0 ATT&CK tactic categories, and 44 campaigns (84.6%) exercise at least one technique in each of the initial-access, execution, discovery, lateral-movement, and exfiltration phases. Only 2 campaigns (3.8%) carry no explicit platform or software signal and therefore rely entirely on tactic-level heuristics for environment inference.
To test whether this platform inference is internally stable rather than arbitrary, we compare three signal sources for each campaign: technique metadata, software-linked platforms, and tactic-implied heuristics. They converge for 96.2% of campaigns. The two divergent cases are Operation MidnightEclipse and RedPenguin, where technique-level platforms define a broad cross-platform compatibility envelope while software-linked evidence points to Linux and tactic-implied heuristics point to Windows. We keep those conflicts explicit rather than collapsing them into a single OS claim.
Software linkage is common at the campaign and intrusion-set layer. The Enterprise bundle contains 693 active malware and 91 tool objects (784 total). Of the 52 campaigns, 47 (90.4%) are linked to at least one software object. Among the 172 active intrusion sets, 158 (91.9%) carry at least one software association.
Specificity is the limiting factor. Only 2.4% of the 784 software objects contain any version indicator, whether in name, aliases, or external_references. CPE identifiers, which directly support version-pinned
installation, appear in only 0.0% of software objects. Software presence is well documented, but software specificity for automated SUT construction is largely absent: most software references remain family-level labels without a version
pin (Figure 3).
At the campaign level, that bottleneck propagates into profile completeness. Structured fields already support coarse environment profiles for 47 of the 52 Enterprise campaigns, which is useful for initial narrowing before analyst refinement, but anchored and exploit-pinned profiles remain rare because they require the version/CPE or campaign-CVE signals that the bundle rarely carries. We next ask whether vulnerability evidence can supply that missing anchor.
Vulnerability evidence is even sparser. CVE identifiers carried by technique, software, campaign, and intrusion-set objects total 26 unique CVEs. Every extracted identifier passes year-range and sequence-length validation, so format-level extraction precision is total; the only systematic miss is the non-canonical “CVE YYYY-NNNN” spacing variant, which in this corpus occurs only inside free-text uses relationship descriptions. Those relationship descriptions are procedure examples, such as a tool exploiting a named CVE, and they annotate how a technique was used rather than which environment a campaign targets. We exclude them from object-level extraction and treat them, like technique-description mentions, as illustrative context rather than as evidence of a structured environment. The bundle contains zero dedicated vulnerability STIX Domain Objects (SDOs), so CVE evidence appears in structured external references or free-text descriptions. Of the 26 CVEs, 14 appear only as illustrative examples in technique descriptions (for example, Exploit Public-Facing Application citing CVE-2016-6662). These are not campaign-specific. The remaining 12 are actionable because they are linked to software, campaign, or intrusion-set objects.
Coverage remains low at the campaign and group level. Only 5 campaigns (9.6%) carry any CVE reference, and only 4 intrusion sets (2.3%) have at least one CVE. The gap, therefore, lies in what becomes machine-readable, not just in what is publicly reported: even when evidence of vulnerability exists, it is split between structured external references and free-text descriptions.
To test how much of this gap is recoverable without changing the corpus, we promote CVE mentions from free-text descriptions into machine-readable CVE slots and then recount coverage. Under this normalization, campaign-level CVE coverage rises from 2/52 (3.8%) to 5/52 (9.6%), a gain of 3 campaigns (+5.8 percentage points). For intrusion sets, coverage rises from 2/172 (1.2%) to 4/172 (2.3%), a gain of 2 (+1.1 percentage points). Figure 4 summarizes the attrition path from all detected CVEs to campaign-level usable evidence.
The gain remains small because most campaigns do not provide explicit evidence of vulnerability. Only 8 actionable CVEs are tied directly to campaign objects in the Enterprise STIX dataset, spanning the five campaigns in Table 6. Four retain linked software and platform evidence and therefore reach the exploit-pinned tier; ShadowRay is the counterexample, carrying CVE-2023-48022 but no linked software object and no software-derived platform signal, so it does not anchor an exploit-pinned lower-bound SUT claim under the conservative rules used in this paper.
| Campaign | CVE(s) | SW | Plat | Tier |
|---|---|---|---|---|
| APT28 Nearest Neighbor Campaign | CVE-2022-38028 | ✔ | Exploit-pinned | |
| Operation MidnightEclipse | CVE-2024-3400 | ✔ | Exploit-pinned | |
| Versa Director Zero Day Exploitation | CVE-2024-39717 | ✔ | Exploit-pinned | |
| SharePoint ToolShell Exploitation | CVE-2025-49704, -49706, -53770, -53771 | ✔ | Exploit-pinned | |
| ShadowRay | CVE-2023-48022 | – | Not anchored |
The contrast between SharePoint ToolShell Exploitation and ShadowRay makes the boundary concrete. SharePoint ToolShell Exploitation reaches the exploit-pinned tier because the bundle carries four campaign-linked CVEs, linked software, and a software-derived platform signal. Even in that strongest case, the structured evidence still stops short of a replay-ready SUT: ATT&CK names vulnerable software and CVEs, but not the exact product-version tuple, trust boundary, or deployment topology needed to automatically instantiate the lab.
ShadowRay fails in the opposite direction. It carries a campaign-level CVE, but without linked software and without a software-derived platform signal, that vulnerability reference cannot be turned into an exploit-pinned lower-bound SUT claim. CVE evidence can only pin down exploitability after software identity and the coarse platform have already been established. Recent automated CVE reproduction pipelines [18] show that once a CVE is fully resolved to a product and exploit, the operational gap can be closed with high coverage; the bottleneck this paper measures sits one layer upstream, on the CTI side.
The concretization audit provides a bounded downstream probe of the remaining gap. When we intentionally narrow the problem to installable package ecosystems such as pip or apt, only one of the eight campaign/CVE pairs reaches
an automatic public path. In the current slice, ShadowRay / CVE-2023-48022 is the only pair that resolves to an open-package target (pip: ray). The remaining seven pairs still terminate in Windows-component,
appliance, or enterprise-server classes that require product-specific reconstruction outside the public path. The automatic path is therefore entirely pip-based; no pair reaches the current apt-materializable path. None of the
eight pairs has a direct ATT&CK target-product binding: the single automatic path comes from curated CVE-plus-package metadata rather than from an ATT&CK software object that already names the vulnerable product.
Across the corpus, 38/52 campaigns (73.1%) are linked to at least one Initial Access technique even though CVE evidence remains sparse. Of those, 33 campaigns (63.5%) lack campaign-level CVE evidence, confirming that Initial Access is often documented without explicit exploit identifiers in the current encoding.
Backend compatibility. RQ2 asks whether the environment signal measured above is strong enough to support backend claims. Our answer is conservative: CF/VMR/ID is a lower-bound backend taxonomy rather than execution ground truth. Among the 691 active Enterprise techniques, 19 (2.7%) are container-feasible, 526 (76.1%) are VMR, and 146 (21.1%) are infrastructure-dependent (Figure [fig:compatibility95distribution]).
Techniques classified as VMR or ID cannot be reproduced faithfully in a container-only environment. Campaigns involving lateral movement, boot-level persistence, or credential access through domain services consistently fall into VMR or ID classes, which sharply limits the coverage of container-based cyber ranges.
The conservative fallback inflates the VMR share, but non-CF techniques remain the majority across all sensitivity settings: they span 29.9% to 97.2% of ATT&CK between the floor and ceiling policies. Without a fallback assignment, explicit rules cover 32.7% of techniques, and the resolved subset is 91.6% in the non-CF class. Container-only replay, therefore, covers only a small minority of ATT&CK under every sensitivity setting. In Figure [fig:compatibility95distribution], the right-side interval is therefore a policy-sensitivity band, not a confidence interval: the floor forces unresolved techniques to CF, the ceiling forces them to ID, and the marker reports the explicit-rule subset before fallback completion.
All 691 techniques receive a deterministic class assignment (CF, VMR, ID). This supports automated backend-family assignment even when faithful replay is not supported. Table 7 details where those assignments come from and shows that the strongest part of the taxonomy is a lower bound on container-feasible coverage, while the weaker part is conservative VMR completion. The per-tactic decomposition preserves the same reading at finer granularity: the non-CF share remains the majority within every tactic family in which techniques appear, so no single ATT&CK tactic rescues container-only replay on its own.
4pt
| Rule evidence | Class | Techniques | Share |
|---|---|---|---|
| Domain, identity, or cloud cue | ID | 132 | 19.1% |
| Lateral-movement tactic fallback | ID | 14 | 2.0% |
| Kernel or boot pattern | VMR | 34 | 4.9% |
| Privileged or kernel name pattern | VMR | 27 | 3.9% |
| Container-compatible targets | CF | 19 | 2.7% |
| Conservative fallback | VMR | 465 | 67.3% |
Most ID assignments come from explicit platform cues already present in ATT&CK, while most VMR assignments come from conservative fallback rather than from a rich machine-readable notion of hypervisor or domain topology. The taxonomy is therefore strongest as a lower bound on container-only coverage: it identifies the small CF minority and the broad non-CF majority, while exact VM layout and multi-host topology remain separate deployment commitments outside the class labels.
To make that reading auditable, we draw both explicit-rule and fallback rows into a class-balanced validation sample: 12 CF, 12 VMR, and 12 ID across 36 rows in total, with 29 explicit-rule rows and 7 fallback rows, so it separately exposes the narrow CF minority, the explicit ID evidence, and the conservative VMR completion. The sample is a construct-validity surface: independent labels can be added to the sampled rows, after which agreement and Cohen’s \(\kappa\) are recomputed over the same audit slice.
The compatibility analysis tells us what kind of lab a technique needs. The next question is whether those observable SUT properties can distinguish one threat actor from another (RQ3).
Profile specificity. Of the 172 active intrusion sets, 90.7% have a unique SUT profile at Jaccard distance \(\delta = 0.1\) when profiles use software constraints only. Adding CVE constraints leaves this value unchanged at 90.7%. That result is consistent with sparse CVE linkage: only 4 intrusion sets carry any CVE evidence. The remaining 9.3% of groups still share their SUT profile with at least one other group.
A SUT built for group \(g\) can therefore still match profiles of other groups within distance \(\delta\), so a successful emulation is not actor-specific by default. The same pattern of ambiguity observed in behavioral attribution studies [13] reappears here at the environmental level. Bootstrap resampling with 5,000 replicates leaves that reading unchanged: confusion remains within 5.2%–14.0%, and the unique-profile rate remains within 86.0%–94.8%. The reading is also flat across the tested near-neighbor thresholds: confusion stays at 9.3% for \(\delta \in \{0.05, 0.10, 0.15, 0.20, 0.30\}\).
The ablations point in the same direction. Adding a CVE, platform, or OS family signal does not materially improve separability beyond what software identity alone provides. Compatibility-summary features add 0.6 pp of confusion reduction. A cardinality-preserving null model (1000 iterations, randomizing software identities while keeping per-IS counts) places the observed confusion inside its expected interval (8.1%–10.5%, permutation \(p=0.34{}\)). The key result is that confusion is largely a low-evidence phenomenon: at \(\delta = 0.10\), it disappears once a group carries at least two linked software items.
These rates are reported over different populations, which we make explicit to avoid misreading. The 9.3% figure is the baseline over all active intrusion sets, and it is concentrated in evidence-poor groups: intrusion sets with little or no linked software have near-empty profiles that collide with one another, so most of the baseline confusion does not reflect genuine environmental overlap among well-evidenced actors. Restricting to the subset of groups with at least one linked software item (\(k \geq 1\)) already lowers confusion to 1.3%. Raising the minimum evidence threshold from one linked software item to two then eliminates profile confusion under \(\delta = 0.10\): confusion drops from 1.3% at \(k \geq 1\) to 0.0% at \(k \geq 2\) and remains collapsed at higher thresholds. These thresholds retain most of the population rather than a sparse tail: of the 172 active intrusion sets, 158 carry at least one linked software item, 132 carry at least two, 105 at least three, and 76 at least five, so the \(k \geq 2\) subset on which confusion vanishes is the majority of the corpus rather than an effect of small samples. Adding CVE evidence does not materially improve separability at that threshold, so the actor-specific screening claim is strictly a function of software-evidence density. The nearest-neighbor cumulative distributions of software-only and software-plus-CVE profiles are almost indistinguishable across the full distance range, which is the visual counterpart of the unchanged 9.3% confusion rate above.
Enrichment headroom. We now quantify the marginal value of the bounded enrichment steps defined above. We report how much additional environment signal becomes machine-actionable when description mining and deterministic compatibility assignment are added on top of structured ATT&CK fields.
Structured ATT&CK fields expose version signal for only 2.4% of active software objects. Description-based regex enrichment raises this to 3.2%, a gain of 0.8 pp. Campaign-level CVE coverage shows a similarly bounded pattern: structured links alone cover 3.8% of campaigns, and free-text CVE extraction raises this to 9.6% (+5.8 pp). Compatibility assignment behaves differently. Explicit rules cover 32.7% of Enterprise techniques, while a deterministic fallback policy resolves 91.6% of non-CF techniques, yielding a larger 58.9 pp increase in backend assignment coverage. The largest measured gain, therefore, comes from deterministic compatibility assignment rather than from version or CVE extraction (Table 8).
2pt
| Signal | Base | Post-step | Gain |
|---|---|---|---|
| Software version signal | % | % | +0.8 pp |
| Campaign-level CVE signal | % | % | +5.8 pp |
| Technique backend assignment | % | % | +58.9 pp |
Taken together, these measurements define the evidence stack available before analyst specification begins (Table 9). Three representative campaigns in Figure 6 show how derivation stops at different missing layers depending on which evidence the corpus carries.
| Evidence layer | Measured signal | What structured CTI already supports | What remains outside the current corpus | ||
|---|---|---|---|---|---|
| Platform coverage | Near-universal platform tags; sparse system-requirement fields | Broad OS, device, or cloud-family narrowing and software-derived compatibility envelopes | Exact OS version, host configuration, deployment placement, and most campaign-specific target declarations | ||
| Software specificity | Campaign and intrusion-set software links are common, but version and CPE signal are rare | Coarse profiles and installable product-family hints; occasional anchored profiles when version/CPE appears | Product-version tuples precise enough for unattended installation and replay-ready package selection | ||
| Vulnerability evidence | Few campaign-linked CVEs; many CVEs remain only in prose | Exploit pinning for a small minority of campaigns when CVE evidence aligns with software and platform signal | Machine-readable exploit surface for most campaigns and direct links from vulnerabilities to deployable targets | ||
| Compatibility classification | CF/VMR/ID backend classes for every Enterprise technique | Deterministic backend-family assignment and a lower bound on container-only coverage | Exact VM count, domain layout, multi-host routing, and service dependencies within the non-CF majority | ||
| Profile specificity | Nearest-neighbor confusion over intrusion-set SUT profiles | A test of whether derived environments are distinctive enough for evidence-filtered actor screening | Actor-specific interpretation under sparse evidence; environmental overlap remains common until software density rises | ||
| Bounded enrichment | Description mining and deterministic fallback assignment | A reproducible lower bound on recoverable signal without changing the underlying corpus | Closure of the gap through downstream extraction alone; the measured gains are limited |
Across these examples, derivation acts as a monotone filter. Platform tags rule out impossible environment families; software links add installable products; campaign-linked CVEs pin exploitability only relative to those products; and compatibility classes choose the minimum backend family. SharePoint ToolShell Exploitation is the strongest structured case in the Enterprise corpus and still stops short of replay-ready derivation. Operation MidnightEclipse shows exploit pinning on thin evidence, while ShadowRay shows that a CVE alone cannot anchor a SUT without supporting software and platform evidence. Analyst specification begins at the first missing layer. A declared step-conditioned overlay applied at execution time remains execution support authored outside the corpus, not new CTI evidence.
Taken together, these measurements define one boundary rather than six separate limitations. Structured CTI supports coarse environment narrowing, deterministic backend-family assignment under conservative rules, and qualified exploit-pinned screening. It does not supply the exact versions, topology, or deployment commitments needed for unattended replay-ready SUT derivation. Actor-specific screening also requires an explicit confusion threshold and enough software density: under this thresholding model, confusion collapses only after an actor carries at least two linked software items. Bounded enrichment recovers the narrow version and vulnerability margins and makes backend commitments explicit, but the analyst-authored remainder remains.
The measurements above show that public structured CTI fixes only part of the executable environment and that emulation authors should separate corpus-supported claims about the environment from analyst-authored lab commitments. This section makes that separation operational and answers RQ4: once the corpus-supported elements are held fixed, does the corpus determine a unique SUT? We answer constructively: it does not.
Definitions. Let \(E\) be the set of materialized SUT elements for a campaign, partitioned by provenance into the fixed (corpus-supported) subset \(\mathrm{corpus}(E)\) and the free subset \(\mathrm{free}(E)=E\setminus\mathrm{corpus}(E)\) (analyst-authored plus tool-concretized). Let \(\mathrm{fp}(E)\) be an invariant fingerprint: the SHA-256 digest of the lexicographically sorted \((\textit{identifier},\textit{value})\) pairs over \(\mathrm{corpus}(E)\), with identifiers keyed by STIX object reference rather than display name. The sort makes the digest order-independent and ensures that free elements never enter it, so two SUTs share a fingerprint exactly when they agree on every corpus-supported element. Two SUTs \(S_1,S_2\) are campaign-equivalent when they preserve every corpus-supported element identically (\(\mathrm{fp}(S_1)=\mathrm{fp}(S_2)\)) and admit execution of the same techniques under the same execution mode. A campaign environment is non-unique when two campaign-equivalent SUTs differ in at least one free element. The claim is existential: two such SUTs establish that the campaign is non-unique.
Construction. A deterministic, seeded generator perturbs only free elements of a campaign’s canonical SUT and re-verifies that \(\mathrm{fp}(\cdot)\) is unchanged; a perturbation that reaches a corpus-supported element changes the fingerprint and is rejected, so compatibility holds by construction rather than by assertion. The per-element provenance tally gives the size of the free region, not the cardinality of the compatible set; non-uniqueness is witnessed constructively, not inferred from those percentages. This is not the familiar observation that an incomplete specification admits many models: the free region is measured per element, the fixed region is pinned by a verifiable invariant fingerprint that makes “the same corpus evidence” precise rather than informal, and compatibility is demonstrated by real execution. The non-uniqueness we report is therefore an operational property of CTI-driven reconstruction, not a restatement of incompleteness. Concretely, the construction certifies a sufficient condition for under-determination: when a campaign has at least one free element with two admissible realizations that preserve every corpus-supported element, and each declared technique’s execution precondition, the corpus does not determine a unique SUT under this fingerprint equivalence.
Measured free region. The provenance surface makes the construction measurable. Across 19 campaign SUT profiles, 133 materialized elements are tagged by source: 32 (24.1%) are corpus-supported, 23 (17.3%) are AutoSUT concretizations of under-specified corpus signals, and 78 (58.6%) are analyst-authored. The split is concentrated by dimension. Platform is fully corpus-supported in this materialized slice (25/25), while topology and staged files are entirely analyst-authored (3/3 and 32/32). Software is mostly analyst-authored (19/20); vulnerability evidence is small but more anchored (2/3 corpus-supported). These counts do not enumerate the compatible family; they quantify where the free region lives and explain why a fixed corpus fingerprint can coexist with multiple deployable environments. We deliberately report this determinacy surface rather than a compatible-family cardinality, because a count would depend on the value domain assigned to each free element: a decoy file, a network topology, and a service realization each range over very different spaces, so collapsing them into one scalar would look precise while hiding that modeling choice. The provenance surface instead remains auditable per profile, recording which dimensions the corpus determines, which AutoSUT concretizes from an implied but unspecified need, and which remain of lab authorship.
Executable witness. The executable witness is a standalone, technique-grounded reference rather than a named ATT&CK campaign: it instantiates Exploit Public-Facing Application (T1190) with the disclosed CVE-2021-41773,
chosen because it runs end to end and so lets us verify compatibility by real execution rather than by assertion. Its fixed region is the vulnerable Apache 2.4.49 build, its exposed port, and the Linux platform; its free region is a single analyst-authored
decoy. Three realizations, the canonical SUT plus two generated variants, share one invariant fingerprint, and each executes the real path-traversal exploit that leaks /etc/passwd, with declared and executed modes identical. The same
corpus-supported core, therefore, admits multiple distinct SUTs that all really execute it.
Structural witness at scale. The named-campaign case is the structural witness. For the ATT&CK Enterprise campaign APT41 DUST, only the Linux platform is corpus-supported, and the remaining elements are free. Variants substitute structurally different services satisfying the same abstract precondition (a remote authenticated service realized as OpenSSH or Dropbear, or a web server as Apache or Nginx) under one unchanged corpus fingerprint. This witness is structural: we report compatibility, not real execution, because the campaign’s Linux substrate cannot honestly execute its Windows-oriented techniques. It shows the breadth of the free region rather than an executable consequence.
Scope. The result is existence, not enumeration: we exhibit at least two campaign-equivalent SUTs per campaign, not the generally unbounded full family. The two witnesses bracket an inverse relationship: a corpus that pins enough to be executed leaves a small free region, whereas a large free region corresponds to campaigns the substrate can only cover behaviorally. This is a measured property of downstream CTI use: the public ATT&CK/STIX bundles we evaluate encode adversary behavior and CTI relationships, not complete executable environments.
The results support one operational claim: structured CTI constrains replay environments but does not uniquely determine them. The measured gaps arise primarily from missing structured fields in public CTI datasets rather than from downstream parsing. The results show a consistent ladder: platform tags and software links often narrow candidate environments; version-anchored software and campaign-linked vulnerability evidence rarely make those profiles replay-ready; and deterministic compatibility assignment mainly improves backend-family inference rather than full reconstruction. Because 67.3% of techniques enter VMR through conservative fallback rather than explicit evidence, the taxonomy is strongest as a lower bound on container-feasible coverage and as a conservative upper bound on how much host or infrastructure support a replay backend may need.
Profile analysis sharpens a second point. A SUT profile that is sufficient for one group is often insufficiently distinctive to justify actor-specific replay claims, which therefore need an explicit profile-confusion threshold. In this corpus, confusion is 1.3% at \(k \geq 1\) and falls to 0.0% at \(k \geq 2\) linked software items. Within this screening model, two linked software items are enough to eliminate profile confusion. The same reading survives the \(\delta\) sensitivity check.
The missing semantics are concrete rather than abstract. Most campaign-linked software still lacks version information and CPE identifiers; 5 of 26 CVEs (19.2%) appear only in natural-language description fields; and the schema carries no
field that directly states whether a technique requires a container, a VM, or a multi-host enterprise topology. A CVE identifier alone should therefore not be mistaken for replay-ready exploit evidence, which is consistent with Schloegel et al.’s finding
that CVE identifiers are often overread as stronger evidence of real-world impact than the identifier alone supports [19].
Those same measurements also bound the next system’s layer. A deliberately narrowed downstream concretization path from campaign-linked CVEs to installable package ecosystems such as apt or pip currently covers only 1 of 8
campaign/CVE pairs and 1 of 5 CVE-positive campaigns. In the current slice, ShadowRay / CVE-2023-48022 is the only pair that resolves to an open-package target (pip: ray). The remaining pairs still terminate
in Windows-component, appliance, or enterprise-server classes that require product-specific reconstruction outside the public path. The measured coverage collapse shows why this layer is worth separating from the core CTI measurement, and why it cannot
replace broader CTI-side environment curation.
Four small schema extensions would measurably reduce specific parts of that gap in the current corpus (Table 10).
| Schema addition | Current coverage | Measured gain |
|---|---|---|
| % software | +0.8 pp | |
| on software | % software | +0.8 pp |
| % rules | +67.3 pp | |
| structured CVE relationships | % campaigns | +5.8 pp |
These are small additions, not a schema redesign or a replay guarantee. would represent installable version intervals without requiring exact versions. would encode CF/VMR/ID directly and remove heuristic backend assignment. Existing STIX constructs do not already carry that meaning: Course-of-Action objects describe response actions, marking definitions describe handling constraints, and ATT&CK custom fields such as expose compatibility or curation metadata rather than a deployability class. Structured CVE relationships would move vulnerability evidence from prose into machine-readable links. The identical +0.8 pp gains for and are expected because the same 6 recovered software objects define both margins. Together, these lower bounds give practitioners, CTI curators, and schema designers a concrete design target: shrink the remaining analyst work through version anchoring, explicit vulnerability structure, and declared backend semantics.
This boundary is also orthogonal to the procedural one. A corpus may encode enough behavioral sequence to support replay planning while still lacking the environment evidence needed to instantiate the lab, and the reverse can hold as well. Treating those layers separately prevents replay results from borrowing fidelity from the wrong dimension. A strong replay claim should therefore name both what procedure re-instantiates from the corpus and what environment layer still comes from explicit lab authorship.
That separation gives a practical taxonomy for replay claims. A procedure-complete but environment-incomplete campaign can support a planner benchmark, but its execution result depends on lab choices outside the corpus. An environment-complete but procedure-incomplete campaign can support target reconstruction, but not a faithful campaign replay. When both layers are incomplete, the output is a CTI-inspired scenario rather than a CTI-derived replay. Only when both layers are corpus-supported should a benchmark claim end-to-end replay from structured CTI. AutoSUT measures the environment axis of that matrix, while prior procedural-gap work measures the action-sequence axis; together they define the evidence budget that a replay claim must disclose.
For practitioners, the operational reading should follow the evidence ladder rather than collapsing it into a single, blunt replay claim. Platform and software evidence are already strong enough to eliminate impossible targets and to choose an OS family or backend family early. Campaign-linked CVEs, when they exist, are better read as exploit-screening anchors than as installation-ready replay proof. A corpus-supported profile may therefore justify a claim such as “VM-backed Windows replay required” without justifying a specific build, patch level, exposure surface, or network layout. CTI producers can reduce analyst burden by version anchoring software objects and exposing campaign-level vulnerability links as first-class fields. Emulation authors and benchmark designers should label which environment commitments are derived from the corpus and which still come from lab authorship. Emulation engineers can treat backend-family assignment as corpus-supported evidence while keeping topology, trust boundary, and host configuration explicit as analyst-authored layers unless the corpus encodes stronger semantics. Under that reading, reproducibility should stop at the strongest layer the corpus actually supports, so that emulations, benchmarks, and replay claims do not inherit precision the corpus never supplied.
Jin et al.show that CTI sharing provides limited operational value partly because shared feeds exhibit inconsistent field usage and low structural completeness [20]. Prior ATT&CK surveys identify sparse software versioning as an open problem but do not quantify it [1], [2], [21]. Broader advanced-persistent-threat (APT) detection surveys likewise emphasize that evaluation depends on representative public datasets and explicit capability criteria [22]. This line of work establishes that structured CTI is unevenly populated, but its primary object is field completeness rather than environment derivability. The missing question is not whether CTI describes behavior, but whether it encodes enough environment evidence to justify the strongest lower-bound SUT claim before any lab is declared.
Provenance-auditing work surfaces the same dependency at the evaluation layer. Inam et al.organize provenance-based system auditing as a capture-to-analysis pipeline and identify reduction and evaluation as persistent bottlenecks [23]. Liu et al.show that endpoint-detection conclusions shift with dataset quality and background activity [24], and Bilot et al.show that inconsistent evaluation practices and benchmark fragility still hinder practical deployment of provenance-based intrusion detection systems [25].
Provenance-graph APT detection systems expose a parallel dependency: operational coverage depends on what telemetry and ATT&CK-linked behaviors the underlying datasets can represent [9]–[12], [26]–[28]. Threat-hunting and investigation systems such as POIROT and Tactical Provenance Analysis likewise align higher-level attack behavior with audit or endpoint detection and response (EDR) records, but they start from observed telemetry rather than from environment derivation [29], [30]. Those works measure behavioral completeness, not whether structured CTI carries enough environment evidence to instantiate the lab in which emulation is supposed to run.
A separate measurement study examines the procedural version of this question: whether ATT&CK-in-STIX encodes enough ordering, preconditions, and translation-ready detail for multi-step replay [31]. This paper isolates the environment version of the problem: whether the same datasets encode enough platform, software, vulnerability, and deployability evidence to build the SUT in which those steps could run. The two questions require different units of analysis and different failure criteria. Procedural sufficiency asks whether a campaign can be replayed as an ordered sequence; environment sufficiency asks whether any compatible lab can be instantiated at all.
Recent CTI extraction pipelines improve structured recovery of behavioral intelligence from reports, including tactics, techniques, and procedures (TTP) extraction and graph construction [32]–[35]. Attribution-oriented work shows that analyst confidence is often bounded by evidence quality rather than by the absence of behavioral hypotheses, and that high-level indicators may still be too coarse for reliable actor attribution [13], [14], [36]. Both lines of work still prioritize what happened over where execution should occur, so they do not provide a measurement framework for SUT derivation quality.
Dataset building, emulation, and cyber-range systems generally begin after the environment has been designed, rather than deriving it from structured CTI [37]. This holds across operator-driven frameworks, automated emulation stacks, ICS testbeds, and recent planning systems guided by large language models [3]–[7], [38]–[41]. Effects Language, for example, gives attack graphs executable coordination semantics, but its target operating environment remains external to the formalism; actions query and modify an environment that has already been supplied [6]. These systems provide execution logic, planning support, or benchmark labs, but they do not derive deployment-ready SUT specifications from CTI, unlike manually authored, fixed, or benchmarked environments. What is missing is a measure of how much of that environmental burden is already carried by structured CTI and how much falls outside the schema. That missing measurement object is the contribution of this paper.
Along four measurement axes, the distinction sharpens: whether a line of work consumes public structured CTI, measures environment evidence, emits a deployability or backend claim, and evaluates SUT profile specificity. STIX and CTI field-quality studies consume public CTI but only read the field population. Extraction and attribution pipelines consume it and yield partial actor distinctiveness, not SUT specificity. Emulation and range systems sometimes consume public CTI but assume a declared lab rather than measuring environment evidence or emitting a backend claim. This paper reports all four from public ATT&CK-style structured CTI: it measures the environment evidence already present, issues lower-bound SUT and backend-family claims, and evaluates profile specificity against sparse-evidence conditions. The distinction is narrow but operational. Nearby work measures behavior quality, recovers behavior from reports, or executes behavior on predeclared labs; this paper measures the environment evidence already present in public structured CTI before any lab recipe, range image, or replay substrate is written.
This paper measures a lower-bound derivability surface for replay-ready SUTs, not total recoverability from the underlying reports. Construct validity is limited because our measurements capture only what is explicitly encoded in STIX fields. Version and configuration information present in source reports but absent from structured fields is therefore out of scope. The bounded-enrichment results should likewise be read as a recoverable machine-readable signal in the public datasets, not as an upper bound on what deeper report-level reconstruction could recover. Internal validity is also constrained by the asynchronous update pipeline between the ATT&CK website and downloadable STIX bundles, which may cause extracted counts to underestimate true association rates. This paper measures one ATT&CK snapshot rather than release-over-release trends, so it does not establish whether environment encoding is improving over time.
External validity is bounded by our focus on public ATT&CK datasets and by campaign-level platform inference via software-linked platform tags. Commercial CTI feeds and community threat-sharing platforms such as MISP or AlienVault OTX may carry richer software, version, or CPE metadata than the public ATT&CK-in-STIX bundles we evaluate. Multi-platform tools can also inflate OS-family counts without proving the exact operating system targeted in a specific incident. We measure what structured fields encode in public bundles, not what richer report-level or non-public CTI could support.
Profile specificity has the same object-level scope. Object-reference keys avoid name-string alias errors inside one STIX object, but they do not merge distinct software objects through external product-family, CPE, Package URL, or vendor-name normalization. The reported profile-confusion rates are therefore reproducible bundle-level rates, not a claim about every product taxonomy an external enrichment layer could impose.
The compatibility taxonomy introduces a second construct validity risk: some Linux-only techniques may still require richer host semantics than the structured fields reveal. We mitigate that risk by keeping CF narrow and defaulting unresolved cases upward to VMR. The stratified validation sample covers 36 techniques (12 CF, 12 VMR, 12 ID; 29 explicit-rule and 7 fallback rows) for independent adjudication. The evaluator computes agreement and Cohen’s \(\kappa\) once the independent labels are filled in. The reported results keep those labels pending and use the sample as a construct-inspection surface rather than as evidence of inter-rater consensus, while explicit-rule coverage and fallback sensitivity show how much of the distribution is rule-driven versus conservative completion.
The concretization path is a bounded materialization probe. Its current public automation narrows the downstream problem to package ecosystems such as pip and apt, so it measures a single materialization slice rather than the
full recoverability of product-level exploit targets, as defined by ATT&CK and external metadata. The same applies to the cross-corpus contrast: Enterprise, Mobile, ICS, CAPEC, and FiGHT are compared using a single extraction pipeline to show what each
bundle makes machine-readable, not to claim that their deployment fidelity is otherwise interchangeable.
The lower-bound claims are deliberately asymmetric. Conservative false VMR assignments and conservative “not anchored” outcomes prevent overstated deployability and may leave some real replay opportunities uncounted. The rates are lower bounds on corpus-supported derivation, not upper bounds on expert reconstruction from underlying reports.
The non-uniqueness result (Section 7) is likewise an existence demonstration. The executable witness has a small free region because the disclosed CVE pins most of the environment, while the large-free-region campaign is demonstrated in a structural manner rather than executed. The claim is constructive non-uniqueness, not a measurement of compatible family size.
This paper measures the environment semantics gap in ATT&CK-style structured CTI: how far the corpus can support the derivation of replay-ready SUTs before analyst specification resumes. Across five datasets, the evidence forms a clear ladder. Platform annotations and software links often narrow candidate environments, but version anchoring, CPE identifiers, campaign-level CVE links, and backend semantics remain too sparse to justify unattended SUT derivation. Profile specificity improves only after enough software evidence accumulates, and the non-uniqueness witness shows why this boundary matters: structured CTI constrains replay environments but does not uniquely determine them. The design target is therefore concrete rather than rhetorical: add version ranges, structured CVE relationships, and explicit compatibility semantics; until then, emulation authors should separate corpus-supported environment claims from analyst-authored lab commitments so replay statements inherit only the precision the evidence supports.
This study analyzes public structured CTI bundles and does not involve human subjects, private operational telemetry, or undisclosed victim data. The main ethical risk is dual use: environment-oriented measurements over CTI could be misread as turnkey replay recipes or used to overstate the extent to which public data already supports emulation by adversaries. We mitigate that risk by keeping the main public path measurement-oriented: it audits the claim surface without synthesizing exploits or executing live intrusion steps.
The optional VM-backed path is likewise bounded. It executes only declared campaign/SUT pairs from the public corpus under local control and does not infer new targets, choose attack steps online, or convert sparse CTI into an autonomous offensive planner. The paper also uses lower-bound language throughout: CVE-linked and compatibility results are reported as bounded-derivation claims, and analyst-authored lab commitments remain explicit whenever the corpus lacks the version, topology, or deployment details needed for replay-ready instantiation.
The authors used Grammarly and ChatGPT for grammar checking, limited editorial revision, and minor formatting assistance for figures and illustrations. All scientific claims, analyses, experiments, visual representations, and conclusions were produced and verified by the authors.
https://github.com/sidneibarbieri/autosut-reproducibility-artifact↩︎