Latent Sculpting for Zero-Shot Generalization: A Manifold Learning Approach to Out-of-Distribution Anomaly Detection


Abstract

A critical vulnerability of supervised deep learning in high-dimensional tabular domains is "generalization collapse": models form precise decision boundaries around known training distributions but fail catastrophically when encountering Out-of-Distribution (OOD) data. To overcome this, we propose Latent Sculpting, a hierarchical, two-stage representation learning architecture designed to enforce explicit structural boundaries prior to density estimation. In the first stage, a Transformer-based tabular encoder is trained using our novel Binary Latent Sculpting loss. This objective explicitly condenses benign network traffic into a dense, low-entropy hypersphere while enforcing a strict geometric minimum-distance margin for anomalous patterns. In the second stage, a Masked Autoregressive Flow (MAF) maps this structurally optimized manifold to calculate exact, probabilistic anomaly thresholds. We evaluate this methodology on the CIC-IDS-2017 benchmark under a rigorous zero-shot protocol, deliberately withholding complex attack classes during training to test true OOD generalization. Averaged across three random initialization seeds to ensure statistical robustness, our framework maintains near-perfect classification on known signatures (F1 = 0.980 ± 0.000) while achieving an overall zero-shot OOD F1-Score of 0.867 ± 0.021 and an AUROC of 0.913 ± 0.010 at an 85th-percentile threshold. Most notably, the model achieves an average recall of 78.7% (peaking at 97.2%) on stealthy "Infiltration" attacks and over 94% on low-volume DoS variations—complex distributional shifts where standard supervised and unsupervised baselines historically suffer near-total detection failure. These empirical results demonstrate that explicitly decoupling topological manifold structuring from probabilistic density estimation establishes a highly stable and scalable defense against zero-day cyber threats.

Out-of-Distribution Detection, Zero-Shot Generalization, Manifold Learning, Network Intrusion Detection, Normalizing Flows, Representation Learning, Zero-Day Attacks.

1 Introduction↩︎

Modern Network Intrusion Detection Systems (NIDS) operating on high-dimensional tabular data face a critical vulnerability: the inability to reliably detect zero-day, Out-of-Distribution (OOD) attacks. While machine learning has advanced signature-based detection, the cybersecurity landscape requires models capable of true zero-shot generalization to identify uncatalogued intrusions [1].

Current methodologies fail to achieve this due to a phenomenon we term “generalization collapse.” Supervised deep learning architectures, including recent tabular Transformers [2], optimize purely for class separation. They construct precise decision boundaries around known distributions but lack strict topological constraints in their latent spaces. Consequently, novel OOD anomalies seamlessly overlap with benign representations, leading to catastrophic overconfidence and near-total detection failure for zero-day threats [3]. Conversely, purely unsupervised anomaly detection models attempt to map the baseline benign distribution to identify deviations. However, without the structural guidance of labeled data during representation learning, these models struggle to resolve the highly nonlinear, multi-modal nature of tabular network traffic, yielding unmanageable false-positive rates.

We hypothesize that resolving generalization collapse requires explicitly decoupling topological manifold structuring from probabilistic density estimation. To this end, we propose Latent Sculpting, a hierarchical, two-stage representation learning architecture. By enforcing explicit structural boundaries prior to density estimation, our framework provides a mathematically stable substrate for isolating zero-day distributional shifts.

Our core contributions are:

  • Binary Latent Sculpting Loss: A novel optimization objective that forces a Transformer-based tabular encoder to aggressively condense benign traffic into a dense, low-entropy hypersphere, while simultaneously enforcing a strict geometric minimum-distance margin for known anomalies.

  • Two-Stage Manifold Density Estimation: The coupling of our structural encoder with a Masked Autoregressive Flow (MAF) [4]. The MAF projects the topologically optimized Stage-1 benign manifold into a tractable probabilistic space, allowing for exact, threshold-based likelihood estimation of novel threats.

  • State-of-the-Art Zero-Shot Performance: Evaluated on the high-dimensional CIC-IDS-2017 benchmark using a rigorous zero-shot protocol (deliberately withholding complex attack classes during training). Averaged across three random initialization seeds, Latent Sculpting maintains near-perfect accuracy on known signatures (\(F1 = 0.980 \pm 0.000\)). Crucially, it achieves an OOD zero-shot \(F1\)-Score of \(0.867 \pm 0.021\) and an AUROC of \(0.913 \pm 0.010\) at an 85th-percentile threshold.

  • Detection of Stealthy Intrusions: The architecture overcomes historic baseline failures on stealthy shifts, achieving an average recall of \(78.7\%\) (peaking at \(97.2\%\)) on Infiltration attacks and over \(94\%\) on low-volume DoS variations.

2 Proposed Methodology↩︎

To address the limitations of standard supervised optimization in high-dimensional domains, we propose a two-stage hierarchical architecture. Our framework explicitly decouples the topological structuring of the latent space from probabilistic density estimation. The pipeline consists of a Tabular Transformer encoder trained via a novel Binary Latent Sculpting Loss, followed by a Masked Autoregressive Flow (MAF) for exact likelihood calculation.

2.1 Structured Latent Space via Tabular Transformers↩︎

Network intrusion data is inherently tabular, lacking the explicit spatial or sequential relationships found in images or text. To capture complex inter-feature correlations, we implement a Tabular Transformer Encoder (illustrated in Fig. 1).

Figure 1: Tabular Transformer Encoder Architecture.

Let the input vector be \(x \in \mathbb{R}^D\), where \(D=71\) represents the distinct network features. Standard neural networks project this entire vector simultaneously. Instead, our architecture treats each feature as a discrete token. A feature embedding layer first projects each scalar feature \(x_i\) into a dense vector representation \(v_i \in \mathbb{R}^{d_{model}}\) (where \(d_{model} = 64\)).

Because tabular features possess strict semantic identities (e.g., index 0 always represents "Duration"), we introduce learnable Positional Embeddings \(P \in \mathbb{R}^{D \times d_{model}}\). Recent literature on deep tabular models [2], [5] demonstrates that injecting positional parameters allows the self-attention mechanism to retain feature identity regardless of input arrangement, effectively acting as a "feature tokenizer." The sequence of embedded tokens is passed through multi-head self-attention layers, and a global average pooling operation collapses the sequence into a single, cohesive representation. A final linear head projects this into the latent vector \(z \in \mathbb{R}^d\) (\(d=64\)).

2.2 Binary Latent Sculpting Loss↩︎

The cornerstone of our architecture is the Binary Latent Sculpting Loss, a novel objective designed to aggressively prevent generalization collapse. Standard Cross-Entropy optimization draws unbounded hyperplanes between classes, leaving vast regions of "negative space" where Out-of-Distribution (OOD) data can trigger high-confidence misclassifications. Our loss function restricts the benign distribution to a mathematically compact manifold while establishing a strict geometric exclusion zone for anomalies.

For a mini-batch of \(N\) latent vectors \(z_i\) and their corresponding binary labels \(y_i \in \{0, 1\}\) (where \(0\) denotes benign and \(1\) denotes anomalous), we dynamically compute the centroid of the benign traffic, \(c_b\):

\[c_b = \frac{1}{N_0} \sum_{i : y_i = 0} z_i\]

where \(N_0\) is the number of benign samples in the batch. For each representation \(z_i\), we calculate its Euclidean distance from the benign centroid: \(d_i = ||z_i - c_b||_2\).

To enforce a structural boundary, we introduce a learnable margin parameter \(m\) (initialized to 5.0) and a learnable temperature scalar \(\tau\) (initialized to 1.0). We transform the distance into a geometric logit \(\hat{l}_i\):

\[\hat{l}_i = \frac{d_i - m}{|\tau|}\]

The primary sculpting force is applied via a Binary Cross-Entropy (BCE) objective over these logits:

\[\mathcal{L}_{cls} = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \log(\sigma(\hat{l}_i)) + (1 - y_i) \log(1 - \sigma(\hat{l}_i)) \right]\]

where \(\sigma\) is the sigmoid function. For benign samples (\(y_i=0\)), the BCE loss minimizes \(\hat{l}_i\), pulling them inside the margin (\(d_i < m\)). For anomalous samples (\(y_i=1\)), it maximizes \(\hat{l}_i\), pushing them explicitly outside the margin (\(d_i > m\)).

To ensure the benign manifold remains maximally dense and tractable for subsequent density estimation, we adapt a compactness penalty inspired by Deep Support Vector Data Description (SVDD) [6]. The tightness force is applied exclusively to benign samples:

\[\mathcal{L}_{compact} = \frac{1}{N_0} \sum_{i : y_i = 0} ||z_i - c_b||_2^2\]

The final objective is controlled by a gravity hyperparameter \(\alpha\) (set empirically to 0.05), balancing structural separation with manifold condensation:

\[\mathcal{L}_{total} = \mathcal{L}_{cls} + \alpha \mathcal{L}_{compact}\]

By mathematically enforcing a low-entropy benign cluster surrounded by a margin of empty space, this formulation guarantees that novel, zero-day anomalies cannot seamlessly overlap with the established benign manifold.

The primary objective of the Binary Latent Sculpting Loss is to enforce a strict topological boundary. By explicitly penalizing known anomalies, the loss function actively repels them from the benign manifold, deliberately creating a distinct structural margin, or negative space. This dynamic forces the network to learn deep, discriminative feature representations of malicious traffic rather than relying on superficial signatures. Empirical validation of this geometric separation is visually demonstrated via a 2D UMAP projection of the latent space in Figure 3.

2.3 Probability Density Estimation↩︎

While the geometric margin effectively isolates known attacks, relying solely on distance for zero-shot detection is insufficient for highly complex OOD data. To address this, Stage 2 of our architecture projects the sculpted benign manifold into a probabilistic space using a Masked Autoregressive Flow (MAF) [4] as illustrated in Fig. 2.

Figure 2: Stage 2 Masked Autoregressive Flow (MAF) and Hierarchical Inference Architecture.

Normalizing flows construct complex probability distributions by applying a sequence of invertible, differentiable transformations \(f_\phi\) to a simple base distribution. Let \(u \sim \mathcal{N}(0, I)\) be a standard multivariate Gaussian. The MAF models the probability density of the benign latent representations \(z_{benign}\) through the change of variables formula:

\[\log p_Z(z) = \log p_U(u) + \sum_{k=1}^{K} \log \left| \det \left( \frac{\partial f_k}{\partial z_{k-1}} \right) \right|\]

Because the Stage 1 objective (\(L_{compact}\)) artificially restricts the benign topology into a continuous, dense hypersphere, the MAF operates under optimal conditions. It uses Masked Autoencoder for Distribution Estimation (MADE) [7] blocks to ensure the autoregressive property without sequentially iterating through dimensions, allowing for highly efficient, exact log-likelihood calculations.

2.4 Hierarchical Inference: Triage and Expert Review↩︎

During deployment, the architecture utilizes a highly computationally efficient hierarchical inference strategy.

Stage 1 (Hardware-Efficient Triage): An incoming flow \(x\) is mapped to \(z\). Its distance \(d = ||z - c_b||_2\) is evaluated against the fully learned Stage 1 margin \(m^*\). If \(d > m^*\), the sample falls into the mathematically defined anomaly zone and is instantly classified as an attack (\(\hat{y} = 1\)).

Stage 2 (Probabilistic Expert Review): If the sample falls within the benign margin (\(d \le m^*\)), it represents either true benign traffic or a highly stealthy zero-day intrusion (e.g., Infiltration). The sample is passed to the MAF, which calculates its exact log-likelihood \(\log p(z)\). The final classification is determined by a fixed threshold \(\gamma\), derived from the \(85^{th}\) percentile of benign validation scores:

\[\hat{y}_{final} = \begin{cases} 1 & \text{if } d > m^* \\ 1 & \text{if } d \le m^* \text{ and } \log p(z) < \gamma \\ 0 & \text{otherwise} \end{cases}\]

This two-tier mechanism helps in deterministic isolation of known threats while reserving intensive probabilistic density estimation for the most difficult edge cases, ensuring real-time scalability.

3 Experimental Setup and Results↩︎

3.1 Dataset and Preprocessing↩︎

To evaluate the zero-shot generalization capabilities of Latent Sculpting, we utilize the CIC-IDS-2017 dataset [1]. This benchmark provides high-dimensional, flow-based network traffic records. To enhance the representational capacity of the Tabular Transformer, we engineered two domain-specific features critical for identifying volumetric and exfiltration-based anomalies:

  1. Bytes per Packet: Calculated as the ratio of Total Length of Forward Packets to Total Forward Packets, capturing payload density.

  2. Packets per Second: Calculated as the ratio of Total Forward Packets to Flow Duration, capturing flow intensity.

A small constant (\(\epsilon = 10^{-6}\)) was added to the denominators to guarantee numerical stability. Following feature extraction, we applied a strict variance-based dimensionality reduction. Columns exhibiting zero variance within the training distribution were dropped to prevent matrix singularities. The remaining continuous features (\(D=71\)) were standardized to zero mean and unit variance. To explicitly prevent data leakage, variance calculation and scaler fitting were performed strictly on the training partition prior to any validation or testing evaluation.

3.2 Zero-Shot Partitioning and Asymmetric Balancing↩︎

To rigorously test Out-of-Distribution (OOD) detection, we employ a strict zero-shot experimental protocol. High-volume and brute-force vectors (e.g., DDoS, DoS Hulk, PortScan) remain in the training distribution. Conversely, a predefined subset of complex, stealthy attacks is entirely withheld during representation learning. The strictly segregated OOD testing classes are: Bot, DoS Slowloris, DoS Slowhttptest, and Infiltration.

Network intrusion datasets inherently suffer from extreme class imbalance. To stabilize the Binary Latent Sculpting Loss, we implement an asymmetric balancing strategy during Stage 1 training. We cap the number of benign training samples to exactly match the frequency of the single most prevalent anomaly (DoS Hulk, \(N=184,804\)), while preserving all available instances of seen anomalies. This yields a diverse training set of 620,283 samples (184,804 benign and 435,479 anomalous), preventing the benign gravity force (\(\alpha\)) from overwhelming the geometric margin optimization. For evaluation, strict 1:1 balancing is enforced for both the internal validation set (\(N=217,740\)) and the final unseen OOD test set, guaranteeing that reported metrics reflect true discriminative power.

3.3 Hyperparameter Configuration↩︎

The architecture was engineered for high parameter efficiency, with the Stage 1 Tabular Transformer requiring only 158,912 trainable parameters. The complete configuration for both the Stage 1 structural encoder and the Stage 2 density estimator is detailed in Table ¿tbl:tab:hyperparameters?. Models were optimized over 10 epochs per stage, utilizing AdamW for Stage 1 (incorporating learnable loss parameters) and standard Adam for Stage 2.

Hyperparameter Configuration for Latent Sculpting
Module Parameter Value
Global Batch Size 512
Input Dimension 71
Stage 1: Internal Vector Size (\(d_{model}\)) 64
Transformer Attention Heads (\(n_{head}\)) 4
Encoder Encoder Layers 3
Feedforward Dimension 256
Latent Dimension (\(z\)) 64
Dropout Rate 0.1
Learning Rate \(5 \times 10^{-4}\)
Weight Decay 0
Epochs 10
Max Gradient Norm 1.0
Total Parameters 158,912
Stage 2: Flow Layers (\(n_{layers}\)) 16
MAF Hidden Dimension 512
Input Dimension (from Stage 1) 64
Epochs 10
Learning Rate \(5 \times 10^{-4}\)

3.4 Computational Efficiency and Real-Time Viability↩︎

A recognized bottleneck of deploying complex generative models like Normalizing Flows within NIDS is the severe computational cost required to continuously monitor high-bandwidth traffic. The hierarchical separation in Latent Sculpting natively eliminates this friction. By utilizing a rigid distance margin in the Stage 1 Tabular Transformer, the architecture instantly filters out the overwhelming majority of known, high-velocity attacks (such as DDoS and DoS Hulk) right at the encoder layer. The computationally demanding MAF in Stage 2 is explicitly reserved as a specialized probabilistic filter triggered only for ambiguous, benign-appearing edge cases, ensuring the overall pipeline remains highly performant and viable for real-world, line-rate implementation.

3.5 Qualitative Analysis of the Sculpted Latent Space↩︎

Before evaluating the quantitative classification metrics, we present a qualitative visualization of the Stage 1 embedding space to verify the geometric properties enforced by the Binary Latent Sculpting Loss. Figure 3 illustrates a 2-Dimensional Uniform Manifold Approximation and Projection (UMAP) of the latent features extracted from our best Stage 1 model (Seed 0) on the internal validation set.

a

Figure 3: 2D UMAP projection of the Stage 1 latent space (Seed 0). The Latent Sculpting architecture successfully condenses benign traffic (blue) into a central manifold while repelling known attack vectors (e.g., DDoS, PortScan) into distinct, isolated clusters..

The visualization confirms the intended topological structuring. Benign network flows (blue) are densely concentrated in a central region, establishing a clear baseline manifold. Conversely, known malicious signatures such as DDoS (orange), PortScan (purple), and DoS Hulk (green) are fractured into distinct, isolated clusters pushed away from the benign center.

Crucially, this projection visually confirms the core mechanism of Latent Sculpting—the deliberate creation of a structural margin, or negative space, between the benign manifold and the known attack vectors. The visible empty regions separating the central benign cluster from the peripheral attack clusters represent this learned boundary. By explicitly repelling known anomalies, the loss function forces the network to learn highly discriminative, deep feature representations of malicious traffic rather than merely memorizing signatures. The model maps the fundamental structural differences of attacks into this expansive negative space.

However, this visualization also perfectly illustrates the inherent vulnerability of relying solely on discriminative boundary models for Out-of-Distribution (OOD) detection. While known attacks are heavily penalized and pushed outward into the negative space, highly novel zero-day attacks that do not share these previously learned geometric features risk falling directly inside the dense benign cluster. This visual evidence underscores the necessity of the Stage 2 Masked Autoregressive Flow (MAF) density estimator; by calculating exact probabilistic likelihoods rather than relying on geometric boundaries, Stage 2 identifies stealthy anomalies that successfully breach the perimeter of the benign manifold.

3.6 Performance on Known and Zero-Day Threats↩︎

To validate the efficacy of the Latent Sculpting architecture, we evaluated the framework across three random initialization seeds (0, 42, 1024). The evaluation contrasts performance on an internal validation set containing known attack signatures against a strictly balanced, unseen Out-of-Distribution (OOD) test set. We report Precision, Recall, F1-Score, Area Under the Receiver Operating Characteristic (AUROC), and Area Under the Precision-Recall Curve (AUPRC).

3.6.1 Validating the Generalization Hypothesis (Stage 1 vs. Stage 2)↩︎

Our core hypothesis posits that standard geometric decision boundaries (Stage 1) are highly effective for known distributions but suffer from generalization collapse when encountering zero-day attacks. The empirical results strongly validate this phenomenon.

Operating in isolation, the Stage 1 Tabular Transformer successfully segregates known attacks, achieving a near-perfect average Anomaly F1-score of \(0.980\) on the internal validation set. However, when exposed to the unseen OOD set, Stage 1 exhibits catastrophic failure on stealthy attacks. For example, across all three seeds, Stage 1 achieves a detection rate (recall) of nearly \(0\%\) on the withheld Bot and Infiltration attacks, as these highly novel malicious vectors perfectly mimic the topological structure of the benign manifold and fall inside the learned margin.

The introduction of the Stage 2 Masked Autoregressive Flow (MAF) at an 85th-percentile (\(\gamma_{85}\)) anomaly threshold fundamentally resolves this vulnerability. By calculating the exact probabilistic likelihood of samples residing within the benign hypersphere, Stage 2 successfully isolates stealthy OOD samples. On average, the full two-stage pipeline achieves an OOD Anomaly F1-Score of \(0.867\) and an impressive average AUROC of \(0.913\), proving that density estimation is critical for zero-shot generalization.

3.6.2 Overall Classification Metrics↩︎

Table ¿tbl:tab:overall95metrics? details the average performance of the full Two-Stage architecture across both validation partitions at the 85th-percentile threshold.

Average Two-Stage Performance (\(\gamma_{85}\)) Across 3 Random Seeds
Dataset Class Precision Recall F1-Score
Internal Benign 1.000 0.813 0.893
(Known) Anomaly 0.843 1.000 0.913
Internal Averages: AUROC: 0.978 \(\vert\) AUPRC: 0.979
Unseen Benign 0.893 0.820 0.850
(OOD) Anomaly 0.833 0.900 0.867
OOD Averages: AUROC: 0.913 \(\vert\) AUPRC: 0.882

The two-stage framework maintains exceptional performance on known threats (average Internal AUROC: \(0.978\)). Crucially, on the zero-day OOD set, it maintains high predictive power with an average AUROC of \(0.913\) and an AUPRC of \(0.882\). While the strict 85th-percentile threshold incurs a slight trade-off in benign recall (increasing false positives), it guarantees an aggressive \(0.900\) average recall for anomalous zero-day threats, which is paramount in intrusion detection.

3.6.3 Per-Attack Zero-Shot Recall Analysis↩︎

To demonstrate the architecture’s robustness against specific zero-day attack vectors, Table ¿tbl:tab:ood95recall? breaks down the detection rate (recall) on the withheld OOD classes across all three initialization seeds. We contrast the Stage 1 isolated performance against the Final Two-Stage pipeline.

OOD Per-Attack Detection Rate (Recall): Stage 1 vs. Final Two-Stage (\(\gamma_{85}\))
OOD Attack Type Seed Stage 1 Final (P85) \(\Delta\) Gain
Infiltration 0 0.0278 0.9722 +0.9444
(Stealthy/Targeted) 42 0.0278 0.4444 +0.4166
1024 0.0000 0.9444 +0.9444
Bot 0 0.0000 0.6536 +0.6536
(Command & Control) 42 0.0000 0.4095 +0.4095
1024 0.0000 0.4100 +0.4100
DoS Slowhttptest 0 0.2968 0.9998 +0.7030
(Low-Volume DoS) 42 0.0178 0.8945 +0.8767
1024 0.0244 0.9444 +0.9200
DoS Slowloris 0 0.7001 0.9991 +0.2990
(Low-Volume DoS) 42 0.1130 0.9822 +0.8692
1024 0.5623 0.9990 +0.4367

The breakdown reveals the profound impact of the MAF density estimation. While Seed 42 experienced lower peak performance due to the inherent stochasticity of manifold structuring, Seeds 0 and 1024 exhibit exceptional stability and discriminative power. Notably, the architecture achieves a peak recall of \(97.2\%\) (Seed 0) and \(94.4\%\) (Seed 1024) on Infiltration attacks—a notoriously difficult vector that mimics standard benign traffic flow. Furthermore, zero-shot detection of low-volume DoS vectors (Slowhttptest and Slowloris) universally exceeds \(89\%\), peaking near \(99.9\%\) across multiple seeds. This confirms that decoupling geometric structuring from likelihood estimation establishes a highly resilient defense against uncatalogued distributional shifts.

3.7 Comprehensive Comparative Analysis with Baseline Literature↩︎

To rigorously evaluate the efficacy of the Latent Sculpting architecture, we benchmark our results against two recent state-of-the-art methodologies evaluated on the CIC-IDS-2017 network intrusion dataset: the flow-based comparative study by Xu and Liu [8] and the packet-level generative safeguard framework by Matejek et al. [9]. To ensure proper attribution, we also reference the foundational algorithms evaluated in the flow-based baseline: Multi-Layer Perceptron (MLP) [10], 1D Convolutional Neural Network (CNN) [11], One-Class Support Vector Machine (OCSVM) [12], and Local Outlier Factor (LOF) [13].

3.7.1 Baseline Experimental Setups and Modalities↩︎

A direct comparison requires contextualizing the fundamental differences in data representation and Out-of-Distribution (OOD) evaluation strategies utilized by each study.

  • Xu and Liu (Flow-Level) [8]: This study utilizes the traditional retroactive, flow-level tabular summaries (e.g., flow duration, packet counts) generated from the CIC-IDS-2017 dataset. The authors evaluate four models: MLP, 1D CNN, OCSVM, and LOF. To assess zero-shot generalization, their experimental setup explicitly withholds three specific attacks from the training data: DoS Slowloris, DoS Slowhttptest, and Bot. Supervised models (MLP, CNN) are trained on 80% of benign traffic and 11 known attacks, whereas the unsupervised models (OCSVM, LOF) are trained exclusively on benign traffic.

  • Matejek et al. (Packet-Level) [9]: Unlike the flow-based baseline, this framework operates at the packet level, taking raw payload bytes and specific header features as input. Their architecture employs an encoder-only Transformer for binary classification, coupled with a RealNVP Normalizing Flow to quantify prediction uncertainty. To evaluate OOD generalization, they employ a "leave-one-out" open-set recognition strategy, wherein they iteratively train 14 separate models—each withholding exactly one attack category to test the network’s ability to flag the excluded attack as novel.

3.7.2 F1-Score and AUROC Comparison↩︎

On known, in-distribution threats, supervised models consistently perform well. Xu and Liu report an overall test F1-score of 0.9446 for the MLP and 0.9160 for the 1D CNN. Our Latent Sculpting architecture achieves comparable excellence on known signatures, maintaining a average 98% Stage 1 F1-Score on stage 1 before averaging a final 85th-percentile (\(\gamma_{85}\)) F1-Score of 0.913 across our three initialization seeds for stage 2, while maintaining a near-perfect AUROC of 0.978 on the internal validation set.

However, the disparity in architectural robustness is exposed during OOD classification. Xu and Liu demonstrate that standard supervised models suffer catastrophic generalization collapse when encountering zero-day attacks, with the MLP and CNN OOD F1-scores dropping to 0.2973 and 0.3218, respectively. Their best-performing unsupervised boundary model, OCSVM, achieved an OOD F1-score of 0.7575. In contrast, our final two-stage Latent Sculpting pipeline (\(\gamma_{85}\)) significantly outperforms these flow-based baselines, achieving an average zero-shot OOD F1-Score of 0.867. Furthermore, our architecture yields an average OOD AUROC of 0.913. This directly competes with the packet-level state-of-the-art established by Matejek et al., who reported AUROC scores between 0.95 and 0.99 for detecting novel inputs utilizing computationally heavy raw-byte models.

3.7.3 Comprehensive Recall Analysis↩︎

To understand the granular behavior of these architectures, we present two comprehensive recall analyses comparing Latent Sculpting against both baselines across all evaluated attack vectors.

1. Comparison with Flow-Based Baselines (Xu & Liu): Table ¿tbl:tab:recall95robust? contrasts our framework against the four models evaluated by Xu and Liu. The results strictly validate our hypothesis regarding topological collapse. Standard discriminative models fail drastically on novel data; the flow-based MLP and CNN achieved a 0.0000 recall on withheld Bot attacks, which perfectly mirrors our isolated Stage 1 encoder’s identical 0.0000 average recall. However, introducing the Stage 2 MAF density estimator rescues this collapse, elevating the zero-shot recall for Bot to 49.10% and achieving \(>94\%\) on low-volume DoS vectors, eclipsing the flow-based baseline models. Furthermore, on highly complex known attacks such as Web XSS, Latent Sculpting vastly outperforms the baselines, improving upon the MLP’s 0.0308 recall to reach an average of 0.9926.

Detection Rate (Recall) Comparison vs. Flow-Based Baselines (Xu & Liu)
Attack Type Nature MLP CNN LOF OCSVM Ours: Stage 1 Ours: Final
(Our Setup) [8] [8] [8] [8] (Avg) (\(\gamma_{85}\) Avg)
Bot Unseen 0.0000 0.0000 0.4680 0.0443 0.0000 0.4910
DoS Slowhttptest Unseen 0.0347 0.0838 0.4150 0.9480 0.1130 0.9462
DoS Slowloris Unseen 0.3675 0.3675 0.3176 0.5733 0.4585 0.9934
Infiltration Unseen 0.0000* 0.5714* 0.8571* 0.8571* 0.0185 0.7870
DDoS Seen 0.9993 0.9996 0.9297 0.6299 0.9987 0.9996
DoS Hulk Seen 0.9921 0.9978 0.7633 0.6886 0.9983 0.9988
DoS GoldenEye Seen 0.9932 0.9995 0.8193 0.7358 0.9986 1.0000
FTP-Patator Seen 0.9987 0.9987 0.6688 0.0132 0.9962 0.9991
Heartbleed Seen 1.0000 0.5000 1.0000 1.0000 1.0000 1.0000
PortScan Seen 0.9995 0.9713 0.5280 0.0095 0.9998 1.0000
SSH-Patator Seen 0.9839 0.9864 0.9915 0.0008 0.8939 0.9045
Web Brute Force Seen 0.1395 0.1860 0.1894 0.0100 0.9311 0.9611
Web SQL Inject Seen 0.2500 0.0000 0.0000 0.0000 0.5833 0.9444
Web XSS Seen 0.0308 0.0231 0.0385 0.0308 0.9614 0.9926
*Note: Infiltration was treated as a Known (Seen) attack during training in the Xu & Liu setup.

2. Comparison with Packet-Level Baseline (Matejek et al.): Table ¿tbl:tab:recall95safeguard? compares our approach against the packet-level architecture. Because Matejek et al. evaluated OOD generalization iteratively, we list their "In-Distribution" recall alongside their corresponding "Out-of-Distribution" recall for when that specific class was withheld. Latent Sculpting, despite utilizing computationally lighter flow-based features, proves highly competitive. When faced with OOD Infiltration attacks, the packet-level safeguard achieved a 7.80% detection rate; our Stage 2 density estimation pipeline identifies the same unseen threat with a 78.70% recall. Similarly, our architecture identifies withheld Bot and DoS Slowhttptest attacks at rates of 49.10% and 94.62%, compared to 9.50% and 52.48% by the packet-level baseline.

Detection Rate (Recall) Comparison vs. Packet-Level Safeguard (Matejek et al.)
Attack Type Nature Matejek et al. (Packet-Level) [9] Ours: Latent Sculpting
(Our Setup) In-Dist OOD (Withheld) Stage 1 (Avg) Final \(\gamma_{85}\) (Avg)
Bot Unseen 0.9424 0.0950 0.0000 0.4910
DoS Slowhttptest Unseen 0.9980 0.5248 0.1130 0.9462
DoS Slowloris Unseen 0.9996 0.1737 0.4585 0.9934
Infiltration Unseen 0.9946 0.0780 0.0185 0.7870
DDoS Seen 1.0000 0.2145 0.9987 0.9996
DoS Hulk Seen 1.0000 0.7529 0.9983 0.9988
DoS GoldenEye Seen 0.9999 0.1995 0.9986 1.0000
FTP-Patator Seen 0.9994 0.0030 0.9962 0.9991
Heartbleed Seen 0.4418 0.0002 1.0000 1.0000
PortScan Seen 0.9856 0.8058 0.9998 1.0000
SSH-Patator Seen 0.9998 0.0027 0.8939 0.9045
Web Brute Force Seen 0.9997 0.9439 0.9311 0.9611
Web SQL Inject Seen 0.7222 0.4678 0.5833 0.9444
Web XSS Seen 0.9959 0.6305 0.9614 0.9926
*Note: Matejek et al. evaluated all attack classes iteratively as both In-Distribution and OOD.

4 Conclusion and Future Directions↩︎

In this paper, we proposed Latent Sculpting, a hierarchical two-stage architecture designed to mitigate generalization collapse in Network Intrusion Detection Systems (NIDS). By explicitly condensing benign traffic into a dense, low-entropy hypersphere and enforcing a geometric margin for known attacks, our framework enables a Masked Autoregressive Flow (MAF) to calculate precise probabilistic thresholds for Out-of-Distribution (OOD) data. Extensive evaluation on the CIC-IDS-2017 dataset demonstrates that this approach significantly outperforms both flow-based and packet-level baselines in zero-shot anomaly detection, successfully identifying stealthy and low-volume zero-day attacks.

Moving forward, our research will focus on two primary directions. First, we plan to transition the Stage 1 structural optimization into a semi-supervised learning paradigm [14], [15]. By leveraging only a small fraction of labeled data alongside vast pools of unlabeled traffic, we can minimize labeling costs while maintaining rigid manifold boundaries. Second, we aim to extend this architecture beyond high-dimensional tabular data to evaluate cross-domain adaptation. By applying the Binary Latent Sculpting objective to continuous domains—such as computer vision, signal processing, and Large Language Models (LLMs)—we aim to test the universality of latent sculpting, thereby providing a generalized, domain-agnostic solution for OOD representation learning.

This work establishes that resilient zero-day detection requires active topological management of the latent space to avoid the generalization collapse inherent in standard deep learning models. By serializing manifold construction with probabilistic density estimation, we provide a robust, data-efficient framework for next-generation network security.Source code: https://github.com/Rajeeb321123/Latent_sculpting_using_two_stage_method

Acknowledgment↩︎

This research received partial funding from the United States Department of Homeland Security (DHS). The views expressed in this article are strictly those of the authors and do not necessarily represent the explicit or implicit policies or endorsements of the DHS. The authors also acknowledge utilizing Google’s Gemini large language model to aid in writing and polishing the manuscript. This AI tool was also employed to help write and refine the PyTorch code utilized for training and assessing our proposed approach. All AI-assisted text and code underwent rigorous human review for correctness, and the authors assume complete accountability for the final published work.

References↩︎

[1]
I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization,” in Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP), 2018, pp. 108–116. [Online]. Available: https://www.scitepress.org/papers/2018/66398/66398.pdf.
[2]
Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko, “Revisiting Deep Learning Models for Tabular Data,” Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 18932–18943, 2021. [Online]. Available: https://arxiv.org/abs/2106.11959.
[3]
D. Hendrycks and K. Gimpel, “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks,” in Proceedings of the International Conference on Learning Representations (ICLR), 2017. [Online]. Available: https://arxiv.org/abs/1610.02136.
[4]
G. Papamakarios, T. Pavlakou, and I. Murray, “Masked Autoregressive Flow for Density Estimation,” Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017. [Online]. Available: https://arxiv.org/abs/1705.07057.
[5]
G. Somepalli, M. Goldblum, A. Schwarzschild, C. B. Bruss, and T. Goldstein, “SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training,” arXiv preprint arXiv:2106.01342, 2021. [Online]. Available: https://arxiv.org/abs/2106.01342.
[6]
L. Ruff et al., “Deep One-Class Classification,” in Proceedings of the 35th International Conference on Machine Learning (ICML), 2018, pp. 4393–4402. [Online]. Available: https://proceedings.mlr.press/v80/ruff18a.html.
[7]
M. Germain, K. Gregor, I. Murray, and H. Larochelle, “MADE: Masked Autoencoder for Distribution Estimation,” in Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015, pp. 881–889. [Online]. Available: https://arxiv.org/abs/1502.03509.
[8]
Z. Xu and Y. Liu, “Robust Anomaly Detection in Network Traffic: Evaluating Machine Learning Models on CICIDS2017,” arXiv preprint arXiv:2506.19877v2, Aug. 2025. [Online]. Available: https://arxiv.org/abs/2506.19877v2.
[9]
B. Matejek, A. Gehani, N. D. Bastian, D. Clouse, B. Kline, and S. Jha, “Safeguarding Network Intrusion Detection Models from Zero-day Attacks and Concept Drift,” Computer Science Laboratory, SRI International, 2024.
[10]
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, no. 6088, pp. 533-536, 1986.
[11]
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
[12]
B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson, “Estimating the Support of a High-Dimensional Distribution,” Neural Computation, vol. 13, no. 7, pp. 1443-1471, 2001.
[13]
M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “LOF: identifying density-based local outliers,” in Proceedings of the 2000 ACM SIGMOD international conference on Management of data, 2000, pp. 93-104.
[14]
J. E. van Engelen and H. H. Hoos, “A survey on semi-supervised learning,” Machine Learning, vol. 109, no. 2, pp. 373–440, 2020.
[15]
Y. Ouali, C. Hudelot, and M. Tami, “An Overview of Deep Semi-Supervised Learning,” arXiv preprint arXiv:2006.05278, 2020.