Adversarial LassoNet: Robust Feature Selection via Stability-Driven Sparse Learning
July 04, 2026
Abstract
Sparse feature selection is critical for high-dimensional machine learning, yet traditional \(\ell_1\)-regularized methods are often brittle under observational noise and spurious correlations, leading to unstable feature supports and degraded generalization. Although adversarial training has been widely used to improve model robustness, its interaction with hierarchical sparse feature selection remains underexplored. In this work, we propose Adversarial LassoNet (AdLNet), a stability-driven sparse feature selection framework that integrates input-space adversarial perturbations with the hierarchical sparsity mechanism of LassoNet. We derive a tractable first-order adversarial approximation under local smoothness assumptions and provide an NTK-inspired spectral analysis to characterize how perturbation-driven training can reduce gradient concentration. Experiments on high-dimensional SERS data, six public benchmark datasets, and ColoredMNIST show that AdLNet maintains competitive sparse-selection performance while improving out-of-distribution robustness by 4.4% and feature support reproducibility by 6.3% under nearly matched support sparsity on ColoredMNIST. On the high-dimensional lung cancer screening dataset, AdLNet achieves a 5.3% test accuracy gain and a 6.0% AUC improvement over vanilla LassoNet. Code and dataset are available at https://github.com/719573/Adversarial-LassoNet.
1 Introduction↩︎
Feature selection is a fundamental task in machine learning, particularly in high-dimensional settings where redundant or noisy variables can substantially degrade predictive performance and generalization [1], [2]. Selecting a compact feature set is especially important when the number of input variables is large and the available sample size is limited [3].
Feature selection remains challenging in realistic settings characterized by observational noise, data heterogeneity, and spurious correlations. These issues are closely related to shortcut learning and dataset shift, where models tend to exploit correlations that do not transfer to deployment settings [4]–[6]. In such settings, some variables may appear predictive because they capture spurious patterns in the training data [7]–[9]. As a result, sparse models optimized only for empirical prediction may select features that fit the training distribution but are unstable under perturbations or distribution shifts. These limitations motivate feature selection methods that are not only predictive, but also stable and robust to spurious correlations.
To address these limitations, we propose Adversarial LassoNet (AdLNet), a stability-driven sparse feature selection framework that integrates local input perturbations into the hierarchical sparsity mechanism of LassoNet. The key idea is to use local adversarial sensitivity as an additional training signal for sparse feature selection. Features that are predictive mainly due to spurious correlations may be more sensitive to local input perturbations, whereas more stable predictive features tend to preserve their contribution under small worst-case input changes. By combining the clean prediction loss with a perturbation-driven stability loss, AdLNet encourages the selected feature subset to be both predictive and locally stable. Importantly, the proposed method preserves the original hierarchical sparse optimization structure of LassoNet and modifies only the training objective through an adversarial stability term.
Our main contributions are threefold:
We propose Adversarial LassoNet (AdLNet), a stability-driven extension of LassoNet that integrates input-space adversarial perturbations into hierarchical sparse feature selection. The proposed objective encourages selected features to be both predictive on clean samples and stable under local worst-case perturbations.
We derive a tractable first-order approximation for local adversarial sensitivity under a local smoothness assumption and incorporate it into the hierarchical proximal optimization pipeline. We further provide an NTK-inspired spectral analysis showing that perturbation-driven training can increase the empirical NTK effective rank when a local spectral growth condition is satisfied.
We conduct experiments on high-dimensional SERS data, six public benchmark datasets, and ColoredMNIST. The results show that AdLNet maintains competitive sparse-selection performance while improving out-of-distribution robustness and feature support reproducibility under spurious correlation shifts.
2 Related Work↩︎
2.1 Sparse Feature Selection and Stability↩︎
Conventional feature selection methods often rely on data-driven \(\ell_1\) regularization. For example, LASSO identifies salient variables by imposing an \(\ell_1\) penalty on model coefficients [10], while subsequent extensions introduce structural constraints to model prior correlations among features [11]. Building on these linear foundations, recent deep sparse models enable representation learning and feature selection to be performed jointly. LassoNet [2], for instance, embeds a global feature selection mechanism into neural network training through hierarchical sparsity constraints. However, these \(\ell_1\)-based paradigms are primarily optimized for in-distribution predictive accuracy, and the stability of the selected features may therefore degrade under distribution shifts.
To improve the stability of sparse selection, several robust optimization techniques have been explored. In linear settings, Adversarial Lasso [12] integrates adversarial training with \(\ell_1\) regularization and uses perturbation constraints to stabilize feature selection. However, its linear formulation limits its applicability to high-dimensional problems with complex nonlinear dependencies. In deep learning, methods such as Robust Sparse Deep Feature Selection [13] reduce model sensitivity by penalizing input-gradient magnitudes. Although such penalties can improve stability to some extent, they are typically isotropic magnitude-based regularizers and may not fully exploit the directional information provided by worst-case perturbations for stable feature selection.
2.2 Perturbation-Based Robustness↩︎
Beyond explicit feature selection, perturbation-based learning has become an important paradigm for improving model robustness. Techniques such as adversarial training [14], input-gradient regularization [9], [15], and Sharpness-Aware Minimization (SAM) [16] encourage local consistency under perturbations and favor flatter, less perturbation-sensitive loss landscapes. Recently, sharpness-aware optimization has also been extended to sparse training; for example, single-step SAM variants have been used to improve the efficiency and accuracy of sparse network training [17]. Perturbation-based robustness is particularly relevant in medical AI, where models are often expected to remain reliable under noise, distribution shifts, and spurious correlations [18], [19].
Nevertheless, many perturbation-based methods primarily target predictor robustness or parameter-level sparsity, such as network pruning, rather than input-level variable selection. As a result, improved robustness of the predictor does not necessarily imply that the selected feature support is concise, interpretable, or reproducible. Although our use of local perturbations is conceptually related to counterfactual explanations [20], the goal of our method is different: we use local perturbations as a training signal to guide stable sparse feature selection, rather than to generate counterfactual samples or make causal intervention claims.
3 Adversarial LassoNet↩︎
3.1 Revisiting LassoNet↩︎
We consider a supervised classification problem with a training set \(\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^{N}\), where \(\mathbf{x}_i\in\mathbb{R}^{d}\) denotes the input feature vector and \(y_i\in\{1,\dots,C\}\) denotes the corresponding class label. The objective is to learn a predictor and a sparse subset of informative features that preserve predictive performance.
LassoNet [2] combines a nonlinear neural network branch with a linear skip connection from the input to the output. Specifically, the prediction function is formulated as: \[f_{\boldsymbol{\theta}}(\boldsymbol{x}) = \boldsymbol{\theta}^\top \boldsymbol{x} + f_\boldsymbol{W}(\boldsymbol{x})\] where \(\boldsymbol{\theta} \in \mathbb{R}^{d \times C}\) denotes the weight matrix of the linear skip connection, and \(f_\boldsymbol{W}(\boldsymbol{x})\) represents a nonlinear feedforward neural network parameterized by \(\boldsymbol{W}\). To impose structured feature sparsity, LassoNet introduces a group-sparsity regularization penalty during the joint optimization of the empirical loss \(\mathcal{L}\), while imposing a hierarchical proximal constraint. The core optimization problem can be precisely formulated as the operator \(\mathrm{HierProx}_{\eta\lambda,\boldsymbol{M}}\) in LassoNet:
\[\begin{align} \min_{\boldsymbol{\theta}, \mathbf{W}} \quad & \mathcal{L}(\boldsymbol{\theta}, \mathbf{W}) + \lambda \sum_{j=1}^{d}\|\boldsymbol{\theta}_j\|_2 \\ \text{s.t.} \quad & \|\mathbf{W}_j^{(1)}\|_\infty \le M \|\boldsymbol{\theta}_j\|_2, \quad j=1,2,\dots,d . \end{align} \label{eq:lassnet}\tag{1}\]
where \(\lambda\) is the regularization parameter controlling the level of sparsity, \(\mathbf{W}_j^{(1)}\) denotes the weight vector connecting the \(j\)-th input feature to the first hidden layer of the network, and \(M > 0\) is a hyperparameter that governs the relative strength of the linear and nonlinear components. The \(\ell_\infty\) constraint ensures a strict hierarchical relationship: an input feature is allowed to possess non-zero weights in the nonlinear branch only if its corresponding linear representative remains active. Consequently, LassoNet integrates feature selection directly into end-to-end network training, performing strictly structured feature sparsification while learning nonlinear representations.
LassoNet in Eq. 1 performs feature selection based solely on empirical predictive performance. However, in high-dimensional yet small-sample regimes, such correlation-driven selection tends to prioritize spurious features that capture incidental correlations in the training distribution. A reasonable assumption is that some spurious features are sensitive to minor input perturbations, yielding unstable and non-reproducible sparse feature supports. To resolve this limitation, we propose a stability-driven method that encourages selected features to be both predictive and robust against worst-case local perturbations.
3.2 Adversarial Perturbation for Stability↩︎
We introduce the concept of adversarial sensitivity to explicitly measure and subsequently suppress feature instability. The adversarial sensitivity is defined for each sample-label pair \((\boldsymbol{x},y)\) as follows: \[S_{\rho}(\boldsymbol{x},y;\theta) := \max_{\|r\|_{p}\le \rho} \left[ \mathcal{L}(f_{\theta}(\boldsymbol{x}+\boldsymbol{r}),y)-\mathcal{L}(f_{\theta}(\boldsymbol{x}),y) \right], \label{eq:local95sensitivity}\tag{2}\] where \(\rho>0\) is the perturbation radius, \(\|\cdot\|_{p}\) denotes the vector \(p\)-norm, and \(\boldsymbol{r} \in \mathbb{R}^d\) denotes a bounded local perturbation. Eq. 2 follows the perturbation-based robustness perspective in adversarial training, where worst-case local input perturbations quantify model sensitivity [9], [14], [15].
A large value of \(S_{\rho}(\boldsymbol{x},y;\theta)\) indicates that the prediction at \(\boldsymbol{x}\) is highly sensitive to small worst-case perturbations. In our framework, we operationalize this stability by deriving the worst-case perturbation \(\boldsymbol{r}^*\), which acts as a stability-aware training signal to guide the sparsity to prune unstable features. Since solving the exact inner maximization in Eq. 2 can be computationally expensive, we approximate it using a local first-order expansion.
Note that we focus on the \(\ell_2\)-bounded case, namely \(p=2\). Extensions to other norms follow from the corresponding dual-norm formulation.
Assumption 1. For fixed \((\theta,y)\), we define the per-example loss \(\ell(\boldsymbol{x}):=\mathcal{L}(f_{\theta}(\boldsymbol{x}),y)\). We assume that \(\ell(\boldsymbol{x})\) is differentiable in a neighborhood of \(\boldsymbol{x}\) and that its input gradient is locally Lipschitz. Formally, there exists a constant \(L_\boldsymbol{x}>0\) such that for all perturbations \(\boldsymbol{r}\) satisfying \(\|\boldsymbol{r}\|_2\le \rho\), \[\|\nabla_\boldsymbol{x} \ell(\boldsymbol{x}+\boldsymbol{r})-\nabla_\boldsymbol{x} \ell(\boldsymbol{x})\|_2 \le L_\boldsymbol{x}\|\boldsymbol{r}\|_2. \label{eq:lipschitz95gradient}\qquad{(1)}\]
Assumption 1 is well-justified for neural networks [14]. Modern activation functions (e.g., ReLU, GELU, sigmoid) are Lipschitz continuous, and their layered composition yields locally Lipschitz input gradients. Under the bounded perturbation constraint \(\|\boldsymbol{r}\|_2 \le \rho\), input \(\boldsymbol{x}\) stays within a compact neighborhood, where the loss gradient maintains uniform smoothness.
Given Assumption 1, the local adversarial sensitivity in Eq.@eq:eq:local95sensitivity admits the first-order approximation as follows: \[S_{\rho}(\boldsymbol{x},y;\theta) = \max_{\|\boldsymbol{r}\|_2\le \rho} \langle \nabla_\boldsymbol{x} \ell(\boldsymbol{x}), \boldsymbol{r}\rangle + O(\rho^2). \label{eq:first95order95sensitivity}\tag{3}\]
By the Cauchy–Schwarz inequality, the linearized inner maximization in Eq. 3 yields both the worst-case local loss increase and the perturbation direction that attains it. Specifically, to maximize the inner product, the optimal perturbation \(\boldsymbol{r}^*\) must align with the direction of the input gradient:
\[\begin{align} \max_{\|\mathbf{r}\|_2\le \rho} \langle \nabla_\mathbf{x}\ell(\mathbf{x}), \mathbf{r}\rangle = \rho\|\nabla_\mathbf{x}\ell(\mathbf{x})\|_2, \\ \mathbf{r}^{*} = \rho \frac{\nabla_\mathbf{x}\ell(\mathbf{x})}{\|\nabla_\mathbf{x}\ell(\mathbf{x})\|_2}, \quad \text{if } \nabla_\mathbf{x}\ell(\mathbf{x})\neq 0 . \end{align} \label{eq:worst95case95perturbation}\tag{4}\] In implementation, we use the stabilized form \[\mathbf{r}^{*} = \rho \frac{\nabla_\mathbf{x}\ell(\mathbf{x})}{\|\nabla_\mathbf{x}\ell(\mathbf{x})\|_2+\epsilon},\] where \(\epsilon>0\) is a small numerical constant introduced for stability. A detailed proof is provided in Appendix 6.
The worst-case perturbation \(\boldsymbol{r}^*\) provides a critical stability signal that can be seamlessly integrated into the feature selection mechanism. By embedding this perturbation into the training loop, the framework is explicitly guided to preferentially prune unstable features, encouraging the selected subset to retain features whose predictive contribution is more stable under local perturbations.
3.3 Stability-Driven Objective and Optimization↩︎
We build a unified stability-driven training objective and preserve the original hierarchical sparse optimization pipeline of LassoNet. We replace the standard ERM loss with the following mixed objective:
\[L_{\mathrm{mix}} = (1-\alpha)\,\underbrace{\mathcal{L}(f_{\theta}(\mathbf{x}),y)}_{\text{predictive fidelity}} + \alpha\,\underbrace{\mathcal{L}(f_{\theta}(\mathbf{x}+\mathbf{r}^{*}),y)}_{\text{adversarial stability}}, \label{eq:mix95loss}\tag{5}\] where \(\alpha\in[0,1]\) controls the balance between clean predictive fidelity and local adversarial stability. Eq. 5 encourages the selection of features that support accurate clean prediction and remain stable under worst-case local perturbations. Rather than modifying the structural sparsification mechanism of LassoNet, our method retains the original hierarchical proximal scheme [2] and refines the training signal. Sparse feature selection is therefore shaped indirectly through stability-aware optimization, rather than through a separate feature-wise stability score.
Optimization: The resulting objective is both non-convex and non-smooth because it combines a deep neural parameterization with the hierarchical sparsity-inducing regularization inherited from LassoNet. We optimize it by alternating between adversarial perturbation construction in the input space and hierarchical proximal updates in the parameter space. Concretely, for each mini-batch, we first compute the input-gradient-based perturbation in Eq. 4 . During the outer parameter update, \(\mathbf{r}^{*}\) is treated as a fixed perturbation constructed from the current mini-batch, following standard first-order adversarial training practice. We then evaluate the mixed objective 5 on clean and perturbed samples, and apply the standard gradient and hierarchical proximal updates. The overall procedure is summarized in Algorithm 1.
3.4 A Local Spectral Condition to Increase Effective Rank↩︎
The stability-driven objective in Eq. 5 , optimized by Algorithm 1, is designed to improve the stability of selected feature supports by incorporating local adversarial perturbations into sparse learning. However, the geometric mechanism behind this stability-driven training effect remains to be characterized. To provide such an interpretation, we turn to the empirical Neural Tangent Kernel (NTK) [21], which captures the similarity of parameter gradients across training samples. Under a local first-order perturbation model, we analyze how adversarial perturbations can affect the spectral distribution of the empirical NTK. In particular, we show that the effective rank of the NTK can increase when the perturbation-induced spectral correction satisfies a positive local growth condition, corresponding to a less concentrated empirical gradient geometry.
Let \(\boldsymbol{\Theta} \in \mathbb{R}^{N \times N}\) denote a non-vanishing symmetric positive semi-definite empirical NTK. We define its effective rank as \[r_{\mathrm{eff}}(\boldsymbol{\Theta}) = \frac{\operatorname{tr}(\boldsymbol{\Theta})^2}{\|\boldsymbol{\Theta}\|_F^2}. \label{eq:eff95rank95def}\tag{6}\] where \(\operatorname{tr}(\boldsymbol{\Theta})\) is the sum of its eigenvalues, and \(\|\boldsymbol{\Theta}\|_F^2\) equals the sum of squared eigenvalues. The ratio \(\operatorname{tr}(\boldsymbol{\Theta})^2/\|\boldsymbol{\Theta}\|_F^2\) quantifies the uniformity of eigenvalue distribution: it reaches the true rank when eigenvalues are perfectly equal, and decreases as eigenvalue concentration grows. Therefore, the metric in Eq. 6 quantifies the intrinsic dimension of a matrix through the uniformity of its eigenvalue distribution and is widely adopted in the analysis of overparameterized models [22].
For perturbed inputs, we assume the empirical NTK admits a local first-order Taylor expansion around the clean kernel \(\boldsymbol{\Theta}\): \[\label{eq:ntk95local95taylor} \begin{gather} \widetilde{\boldsymbol{\Theta}}(\rho) = \boldsymbol{\Theta} + \rho \boldsymbol{E} + \boldsymbol{R}(\rho), \\ \text{where} \quad \|\boldsymbol{R}(\rho)\|_F = o(\rho) , \end{gather}\tag{7}\] with \(\boldsymbol{E}\) being a symmetric first-order correction matrix, \(\boldsymbol{R}(\rho)\) denoting the higher-order residual term of the Taylor expansion, and \(o(\rho)\) representing the little-o notation. We assume \(\widetilde{\boldsymbol{\Theta}}(\rho) \succeq 0\) holds for sufficiently small \(\rho > 0\), and let \(\langle \boldsymbol{A}, \boldsymbol{B} \rangle = \operatorname{tr}(\boldsymbol{A}^\top \boldsymbol{B})\) denote the Frobenius inner product throughout.
Lemma 1. Under the above conditions, the effective rank of the perturbed NTK admits the following first-order Taylor expansion with respect to \(\rho\): \[r_{\mathrm{eff}}(\widetilde{\boldsymbol{\Theta}}(\rho)) = r_{\mathrm{eff}}(\boldsymbol{\Theta}) + 2\,r_{\mathrm{eff}}(\boldsymbol{\Theta}) \underbrace{\left[ \frac{\operatorname{tr}(\boldsymbol{E})}{\operatorname{tr}(\boldsymbol{\Theta})} - \frac{\langle \boldsymbol{\Theta}, \boldsymbol{E}\rangle}{\|\boldsymbol{\Theta}\|_F^2} \right]}_{G(\boldsymbol{E};\boldsymbol{\Theta})}\rho + o(\rho). \label{eff95rank95expansion}\qquad{(2)}\]
The term \(\operatorname{tr}(\boldsymbol{E})/\operatorname{tr}(\boldsymbol{\Theta})\) in ?? measures the trace-normalized spectral mass added by the perturbation, whereas \(\langle \boldsymbol{\Theta}, \boldsymbol{E}\rangle / \|\boldsymbol{\Theta}\|_F^2\) measures how strongly the correction aligns with the energy already concentrated in the clean NTK. Thus, the effective rank increases when the perturbation contributes relatively more mass to weak or underrepresented directions than to the dominant kernel directions.
Lemma 1 shows that the first-order variation of the effective rank is governed by \(G(\boldsymbol{E};\boldsymbol{\Theta})\), which we refer to as the spectral growth score.
Proposition 2 (Local condition for effective-rank increase). If \[G(\boldsymbol{E};\boldsymbol{\Theta}) > 0,\] then there exists \(\rho_0 > 0\) such that for all \(0 < \rho < \rho_0\), \[r_{\mathrm{eff}}(\widetilde{\boldsymbol{\Theta}}(\rho)) > r_{\mathrm{eff}}(\boldsymbol{\Theta}).\]
Proof. From Lemma 1, the change in effective rank is given by
\[\Delta r_{\mathrm{eff}} = 2 r_{\mathrm{eff}}(\boldsymbol{\Theta}) G(\boldsymbol{E};\boldsymbol{\Theta})\rho + o(\rho).\] Since \(r_{\mathrm{eff}}(\boldsymbol{\Theta}) > 0\) for a non-vanishing NTK, the condition \(G(\boldsymbol{E};\boldsymbol{\Theta}) > 0\) implies the linear coefficient is strictly positive. By the definition of the little-o notation, the linear term strictly dominates the remainder \(o(\rho)\) for a sufficiently small \(\rho > 0\), guaranteeing \(r_{\mathrm{eff}}(\widetilde{\boldsymbol{\Theta}}(\rho)) > r_{\mathrm{eff}}(\boldsymbol{\Theta})\). ◻
The theoretical condition \(G(\boldsymbol{E};\boldsymbol{\Theta})>0\) derived above provides a local mechanism for effective-rank elevation. Interested readers may refer to Appendix 7 for details.
4 Experiments↩︎
4.1 Experimental Setup↩︎
Datasets. We conduct experiments on three groups of datasets: the Surface-Enhanced Raman Spectroscopy (SERS) lung cancer screening dataset, six public benchmark datasets, and the ColoredMNIST benchmark.
SERS lung cancer screening dataset. The SERS dataset is used to evaluate sparse feature selection in a high-dimensional medical classification scenario. It contains 87 samples, including 47 positive and 40 negative cases, and each sample consists of 1,264 spectral variables. The dataset is randomly split into training, validation, and test sets with a ratio of 70%/10%/20%. This setting is consistent with recent Raman spectroscopy studies, where machine-learning models are used to classify disease-related spectra and identify diagnostically informative spectral patterns [23], [24]. All spectral variables are standardized using the statistics of the training set.
Public benchmark datasets. We further evaluate predictive performance on six widely used public benchmark datasets, including MNIST [25], MNIST-Fashion [26], ISOLET [27], COIL-20 [28], Activity [29], and Mice Protein [30]. These datasets cover image recognition, speech recognition, human activity recognition, and biological classification tasks, providing diverse feature distributions and data complexities for evaluating sparse feature selection. For image datasets, pixel values are normalized to \([0,1]\); for non-image datasets, continuous features are standardized using training-set statistics.
ColoredMNIST benchmark. To evaluate robustness under spurious correlation shifts, we adopt ColoredMNIST [31], a controlled benchmark in which color is spuriously correlated with class labels. Following standard protocols, the training environments are constructed with color-label correlations of \(\{0.1,0.2,0.3\}\), and the shifted test environment is used to evaluate out-of-distribution robustness and feature support reproducibility.
Experimental settings. All experiments are implemented in PyTorch 1.8.0 with CUDA 11.1 and conducted on a workstation equipped with an Intel(R) i7-7700 CPU @ 3.60GHz, 32 GB RAM, and an NVIDIA GeForce RTX 3080 GPU. All models are trained from scratch without pretraining. We use stochastic gradient descent (SGD) with momentum \(0.9\), and the learning rate is set to \(10^{-3}\). Batch sizes are selected according to dataset scale, ranging from \(128\) to \(512\). The network configurations and training epochs follow the corresponding baseline settings. Unless otherwise specified, experiments are repeated over five independent random seeds.
Vanilla LassoNet [2] is used as the primary baseline because it shares the same hierarchical sparsity mechanism as AdLNet but does not include adversarial stability regularization. Therefore, the comparison between AdLNet and vanilla LassoNet directly reflects the effect of the proposed stability-driven objective. On the six public benchmark datasets, we further compare with FISTA-Net [32] and Deep-Lasso [33], which represent optimization-driven sparse learning and gradient-regularized deep feature selection, respectively.
For the six public benchmark datasets, all methods are evaluated under the matched sparse feature budget of \(k=50\) following the original LassoNet regularization-path protocol [2]. Specifically, we vary the sparsity regularization parameter \(\lambda\) and report the result at the checkpoint satisfying the target feature budget. For the SERS dataset, the main result is reported under the constraint \(n_{\mathrm{selected}}<300\) to retain sufficient spectral coverage for medical classification, and an additional controlled sparse-budget analysis is conducted to examine the influence of feature-count differences. For ColoredMNIST, methods are compared under nearly matched support sparsity, so that robustness and reproducibility improvements cannot be attributed to selecting more features.
The adversarial mixing coefficient \(\alpha\) controls the trade-off between predictive fidelity and perturbation-driven stability. To avoid data leakage, \(\alpha\) is selected using a validation-only protocol. The candidate values are \(\{0,0.2,0.4,0.6,0.8,1.0\}\) and are evaluated only on the validation set. The model checkpoint with the best validation accuracy is then evaluated on the held-out test set. Other key hyperparameters, including the hierarchy parameter \(M\) and the sparsity regularization path, are selected using the same validation protocol. For AdLNet, adversarial perturbations are generated using the first-order approximation derived in Section 3.2, and optimization follows Algorithm 1.
Evaluation Metrics. We evaluate the proposed method from three complementary perspectives: predictive performance, robustness and support reproducibility under distribution shift, and optimization-geometry diagnostics.
For the six public benchmark datasets, we report classification accuracy under the matched sparse feature budget. For the SERS medical dataset, we additionally report Sensitivity (Sens), Specificity (Spec), and area under the ROC curve (AUC), which provide a more comprehensive evaluation of binary medical classification performance.
For ColoredMNIST, we report in-distribution (ID) accuracy, out-of-distribution (OOD) accuracy, and the ID–OOD generalization gap to assess robustness under spurious correlation shifts. Feature support reproducibility is quantified using the Jaccard index across independent runs, measuring the consistency of selected feature subsets under different random initializations.
For training-dynamics analysis, we compute empirical NTK effective rank, dominant eigenvalue ratio, spectral energy concentration, and Hessian curvature statistics. These metrics are used as diagnostic evidence to examine whether stability-driven training reduces spectral concentration and sharp curvature in sparse learning. Detailed metric definitions and calculation procedures are provided in Appendix 8.
4.2 Predictive Performance under Sparse Feature Selection↩︎
We first evaluate whether the proposed stability-driven objective can preserve predictive performance under sparse feature selection. This evaluation is conducted on the high-dimensional SERS medical dataset and six public benchmark datasets. The SERS experiment examines the behavior of AdLNet in a small-sample, high-dimensional regime, while the public benchmark experiments further test whether the method remains effective across diverse data domains under matched feature budgets.
Table 1 reports the results on the SERS lung cancer screening dataset. Compared with vanilla LassoNet, AdLNet achieves better test performance under validation-selected sparse checkpoints, particularly in terms of test accuracy and AUC. This result suggests that introducing adversarial stability into the sparse learning objective can improve out-of-sample generalization in the high-dimensional medical setting, where prediction-only sparse fitting may be more sensitive to unstable correlations in limited training data.
To further examine whether this improvement simply comes from selecting more features, we conduct a controlled sparse-budget analysis in Table 7. Under a high-sparsity regime (\(k \approx 120\)), AdLNet slightly underperforms vanilla LassoNet, indicating that excessive stability pressure may remove some predictive but perturbation-sensitive features and lead to underfitting. In contrast, under a moderate-sparsity regime (\(k \approx 250\)), AdLNet achieves higher test accuracy than vanilla LassoNet. These results reveal an important trade-off between sparsity and stability: adversarial perturbation is most beneficial when the feature budget is sufficient to retain stable predictive features, while overly restrictive sparsity may limit its advantage.
Table 2 further reports the results on six public benchmark datasets under the matched feature budget of \(k=50\). AdLNet achieves the best test accuracy on MNIST, MNIST-Fashion, COIL-20, and Activity, and remains competitive on ISOLET and Mice Protein. Therefore, the proposed objective does not sacrifice sparse-selection performance on standard benchmark datasets. Together, the SERS and public benchmark results demonstrate that AdLNet maintains competitive predictive performance under sparse feature selection, while offering improved generalization in the high-dimensional medical setting.
| Metric | LassoNet | AdLNet |
|---|---|---|
| Selected Features | 119 | 256 |
| Validation Accuracy (%) | 75.7 | 73.6 |
| Test Accuracy (%) | 72.1 | 76.4 |
| Sensitivity (%) | 85.7 | 71.4 |
| Specificity (%) | 77.8 | 82.5 |
| AUC | 0.775 | 0.835 |
| Dataset | Vanilla LassoNet | FISTA-Net | Deep-Lasso | AdLNet |
|---|---|---|---|---|
| Mice Protein | 90.2 \(\pm\) 2.2 | 94.2 \(\pm\) 2.7 | 91.2 \(\pm\) 1.6 | 91.9 \(\pm\) 2.2 |
| MNIST | 88.5 \(\pm\) 0.9 | 89.0 \(\pm\) 11.3 | 86.0 \(\pm\) 1.6 | 89.4 \(\pm\) 0.9 |
| MNIST-Fashion | 77.3 \(\pm\) 1.0 | 76.3 \(\pm\) 5.7 | 78.9 \(\pm\) 0.6 | 79.0 \(\pm\) 1.7 |
| ISOLET | 82.0 \(\pm\) 1.1 | 85.4 \(\pm\) 2.9 | 82.7 \(\pm\) 1.6 | 83.9 \(\pm\) 1.0 |
| COIL-20 | 98.5 \(\pm\) 0.9 | 96.5 \(\pm\) 2.7 | 98.3 \(\pm\) 0.8 | 99.2 \(\pm\) 0.6 |
| Activity | 92.9 \(\pm\) 0.4 | 92.6 \(\pm\) 0.8 | 93.0 \(\pm\) 0.2 | 94.5 \(\pm\) 0.4 |
4.3 Robustness and Support Reproducibility under Spurious Correlation Shifts↩︎
We next evaluate whether the proposed adversarial stability objective improves robustness and feature support reproducibility under spurious correlation shifts. To this end, we conduct experiments on ColoredMNIST, a controlled benchmark in which color is spuriously correlated with class labels during training but becomes unreliable in the shifted test environment. This setting directly tests whether a sparse feature-selection method relies on shortcut-sensitive features or selects features that remain predictive under distribution shift.
Table 3 reports the results. Compared with vanilla LassoNet, AdLNet achieves better out-of-distribution (OOD) accuracy while maintaining nearly identical support sparsity (\(0.969\) versus \(0.966\)). This indicates that the robustness gain is not caused by selecting a denser feature subset. Instead, this improvement suggests that the stability-driven training signal introduced by adversarial perturbation contributes to better OOD robustness. Vanilla LassoNet performs sparse selection mainly according to prediction loss on the training distribution, and therefore may retain shortcut-sensitive features that exploit the color-label correlation. In contrast, AdLNet incorporates local worst-case perturbations into the hierarchical sparsity framework, encouraging the proximal updates to preserve features whose predictive contribution is more stable under local input perturbations.
In addition to OOD robustness, AdLNet also improves feature support reproducibility, as reflected by the higher Jaccard index across independent runs. This suggests that adversarial stability regularization makes the selected feature subsets less sensitive to random initialization and spurious training correlations. Overall, the ColoredMNIST results support our core hypothesis that integrating adversarial perturbation into sparse feature selection can improve both robustness and support reproducibility under spurious correlation shifts.
| Method | ID Acc (%) | OOD Acc (%) | Gap (%) | Support Sparsity | Jaccard |
|---|---|---|---|---|---|
| LassoNet | 84.6 \(\pm\) 0.6 | 47.3 \(\pm\) 4.9 | 37.3 \(\pm\) 4.3 | 0.969 | 0.456 |
| AdLNet | 85.5 \(\pm\) 1.0 | 51.7 \(\pm\) 2.7 | 33.8 \(\pm\) 1.7 | 0.966 | 0.519 |
4.4 Spectral Diagnostics of Optimization Geometry↩︎
We further analyze the empirical NTK and Hessian spectra to examine whether stability-driven training changes the optimization geometry of sparse learning. The local analysis in Section 3.4 suggests that adversarial perturbation can increase the effective rank of the empirical NTK when the first-order spectral growth condition holds. Since this condition is derived for local infinitesimal perturbations, while our experiments compare trained models, the following analysis should be interpreted as an empirical diagnostic rather than a direct proof of the local condition.
To quantify the macroscopic redistribution of NTK spectral mass, we compute the following empirical spectral-redistribution score: \[\hat{G}_{\mathrm{macro}} = \frac{\mathrm{tr}(\boldsymbol{\Theta}_{\rho}-\boldsymbol{\Theta}_{0})}{\mathrm{tr}(\boldsymbol{\Theta}_{0})} - \frac{\langle \boldsymbol{\Theta}_{0},\boldsymbol{\Theta}_{\rho}-\boldsymbol{\Theta}_{0}\rangle}{\|\boldsymbol{\Theta}_{0}\|_F^2},\] where \[\boldsymbol{\Theta}_{0}=J(\mathbf{X};\boldsymbol{\theta}_{0})J(\mathbf{X};\boldsymbol{\theta}_{0})^\top, \qquad \boldsymbol{\Theta}_{\rho}=J(\mathbf{X};\boldsymbol{\theta}_{\rho})J(\mathbf{X};\boldsymbol{\theta}_{\rho})^\top.\] Here, \(\boldsymbol{\theta}_{0}\) denotes the vanilla LassoNet checkpoint, and \(\boldsymbol{\theta}_{\rho}\) denotes the AdLNet checkpoint. Both empirical NTKs are evaluated on the same clean inputs \(\mathbf{X}\), so that the comparison reflects differences induced by the learned parameter configurations rather than by different input samples. A positive value of \(\hat{G}_{\mathrm{macro}}\) suggests that stability-driven training redistributes spectral mass toward less dominant NTK directions at the macroscopic level.
In our experiment, we obtain \(\hat{G}_{\mathrm{macro}}=0.121\), indicating a positive empirical spectral-redistribution trend. As shown in Table 4, AdLNet produces a less concentrated NTK spectrum than vanilla LassoNet, with a higher effective rank and lower dominant-eigenvalue energy concentration. This suggests that adversarial stability regularization suppresses excessively dominant optimization directions and encourages a more distributed gradient geometry under sparse learning.
A similar trend is observed in the Hessian spectrum. Compared with vanilla LassoNet, AdLNet yields a smoother local curvature profile with weaker sharp directions, indicating improved local optimization stability under perturbation-aware sparse learning. Together, the NTK and Hessian diagnostics are consistent with the spectral interpretation in Section 3.4: stability-driven training tends to reduce spectral concentration and increase the effective rank of the empirical NTK. However, these results should be viewed as finite-sample diagnostic evidence rather than a strict verification of the local condition \(G(\boldsymbol{E};\boldsymbol{\Theta})>0\).
| Spectrum | Metric | LassoNet | AdLNet |
|---|---|---|---|
| NTK | Trace \(\mathrm{Tr}(\Theta)\) | 15018.42 | 10219.46 |
| Effective Rank \(\uparrow\) | 7.07 | 8.05 | |
| \(\lambda_1 / \mathrm{Tr}(\Theta) \downarrow\) | 0.343 | 0.310 | |
| Top-10 Energy Ratio \(\downarrow\) | 0.740 | 0.722 | |
| Spectral Growth Coefficient \(\hat{G}_{\mathrm{macro}} \uparrow\) | – | 0.121 | |
| Hessian | Maximum Eigenvalue \(\lambda_{\max} \downarrow\) | 14.08 | 10.13 |
| Trace \(\mathrm{Tr}(H) \downarrow\) | 50.10 | 42.66 | |
| Average Curvature \(\bar{\kappa} \downarrow\) | 0.805 | 0.726 |
4.5 Component-wise Ablation Study↩︎
We conduct ablation experiments on the Mice Protein dataset to examine the contribution of each component in AdLNet. Specifically, we compare the full model with vanilla LassoNet and several partial variants, including LassoNet with only the stability-driven loss, LassoNet with adversarial perturbation, and LassoNet with a gradient-norm penalty replacing directional adversarial perturbation.
As shown in Table 5, the full AdLNet model achieves the best overall performance, with the highest test accuracy (\(92.0\%\)) and the highest feature support consistency measured by the Jaccard index (\(0.71\)). Compared with the partial variants that use only the stability-driven loss or only adversarial perturbation, the full model achieves better predictive performance and support reproducibility. This indicates that the observed improvement cannot be explained by either component alone, but arises from their joint integration within the hierarchical sparsity framework.
Replacing the directional adversarial perturbation with a gradient-norm penalty also degrades performance and introduces additional computational cost. This result suggests that the key benefit of AdLNet comes from explicitly probing locally vulnerable input directions, rather than simply imposing a generic smoothness regularization. In other words, the directional perturbation provides a more informative stability signal for guiding sparse feature selection.
Overall, the ablation results demonstrate that robust prediction and reproducible feature selection rely on the joint effect of perturbation-based stability regularization and hierarchical sparse support formation.
| Method | Val Acc (%) | Test Acc (%) | Jaccard | Selected Ratio (%) | Time (s) |
|---|---|---|---|---|---|
| LassoNet | 89.8 \(\pm\) 2.5 | 90.6 \(\pm\) 3.7 | 0.66 | 34.8 | \(215.3 \pm 1.9\) |
| LassoNet + Stability Loss | 91.3 \(\pm\) 2.7 | 90.4 \(\pm\) 2.7 | 0.69 | 34.5 | \(305.6 \pm 11.5\) |
| LassoNet + Adversarial | 90.6 \(\pm\) 2.0 | 91.5 \(\pm\) 3.6 | 0.67 | 34.8 | \(311.8 \pm 20.0\) |
| LassoNet + Gradient Norm | 90.1 \(\pm\) 3.1 | 90.6 \(\pm\) 3.6 | 0.65 | 33.8 | \(552.8 \pm 60.6\) |
| Full model | 91.5 \(\pm\) 1.8 | 92.0 \(\pm\) 1.2 | 0.71 | 37.6 | \(472.4 \pm 5.6\) |
4.6 Sensitivity to the Mixing Coefficient↩︎
We finally analyze the effect of the adversarial mixing coefficient \(\alpha\), which controls the balance between predictive fidelity and perturbation-driven stability. As shown in Table 6, AdLNet maintains relatively stable performance across a broad range of \(\alpha\) values on most datasets, indicating that the proposed framework is not overly sensitive to a narrowly tuned mixing coefficient.
The results also reflect the trade-off introduced by the mixed objective. A smaller \(\alpha\) places more emphasis on the original prediction objective, while a larger \(\alpha\) increases the influence of perturbation-driven stability. Therefore, moderate values of \(\alpha\) can better balance clean predictive information and stability regularization in many cases. The optimal value of \(\alpha\) varies across datasets, suggesting that this balance is data-dependent.
| Value of \(\alpha\) | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-3 (lr)4-5 (lr)6-7 (lr)8-9 (lr)10-11 (lr)12-13 Dataset | Val | Test | Val | Test | Val | Test | Val | Test | Val | Test | Val | Test |
| Mice Protein | 91.3 | 90.4 | 92.0 | 91.6 | 90.2 | 90.6 | 90.6 | 91.8 | 91.3 | 90.8 | 92.2 | 90.9 |
| MNIST | 89.1 | 89.0 | 89.4 | 89.2 | 88.5 | 88.4 | 88.8 | 88.0 | 88.9 | 88.6 | 87.8 | 87.7 |
| MNIST-Fashion | 78.1 | 78.2 | 79.3 | 79.1 | 78.8 | 79.4 | 79.0 | 79.6 | 79.0 | 78.7 | 78.5 | 79.1 |
| ISOLET | 83.2 | 83.2 | 84.4 | 84.2 | 83.4 | 83.5 | 84.6 | 83.7 | 82.7 | 82.5 | 82.6 | 82.1 |
| COIL-20 | 99.0 | 99.4 | 99.6 | 99.2 | 98.8 | 99.7 | 98.5 | 99.4 | 98.8 | 99.2 | 98.9 | 99.2 |
| Activity | 98.1 | 92.2 | 98.0 | 93.3 | 98.2 | 93.4 | 98.4 | 93.5 | 98.2 | 93.5 | 98.1 | 93.5 |
5 Conclusion↩︎
This paper presents AdLNet, a stability-driven sparse feature selection framework that integrates local adversarial perturbations into the hierarchical sparsity mechanism of LassoNet. By combining predictive relevance with local input stability, AdLNet encourages selected features to remain informative under both clean inputs and local worst-case perturbations, while preserving the original sparse optimization structure. We derive a first-order approximation for the adversarial perturbation term and provide an NTK-inspired spectral analysis to characterize the effect of stability-driven training on sparse learning. Empirical results on high-dimensional SERS data, six public benchmark datasets, and ColoredMNIST show that AdLNet maintains competitive sparse-selection performance while improving robustness under spurious correlation shifts and enhancing feature support reproducibility. NTK and Hessian diagnostics further provide finite-sample evidence consistent with the proposed spectral interpretation, suggesting that the observed robustness improvements are associated with reduced spectral concentration and a milder local curvature profile.
The current study focuses on feature-level sparse selection within the LassoNet framework and analyzes stability mainly from the perspective of local input perturbations. This setting allows us to clearly examine the interaction between hierarchical sparsity and adversarial stability, but broader forms of sparse learning remain to be explored. Future work will investigate whether the same stability-driven principle can be extended to other sparsity structures, such as weight sparsity and attention sparsity, as well as to more diverse distribution-shift scenarios.
Appendix↩︎
6 First-Order Approximation of Local Adversarial Sensitivity↩︎
This appendix provides a rigorous error bound for the first-order approximation used to construct local adversarial perturbations in the main text. For fixed parameters and label \((\theta, y)\), define the loss function as \[\ell(\boldsymbol{x}) := \mathcal{L}(f_{\theta}(\boldsymbol{x}), y).\] Under Assumption 1 in the main text, \(\ell(\boldsymbol{x})\) is differentiable in a neighborhood of \(\boldsymbol{x}\), and the input gradient is locally Lipschitz continuous. That is, there exists \(L_\boldsymbol{x} > 0\) such that for all perturbations \(r\) with \(\left\lVert r \right\rVert_2 \le \rho\), \[\left\lVert \nabla_\boldsymbol{x} \ell(\boldsymbol{x}+\boldsymbol{r}) - \nabla_\boldsymbol{x} \ell(\boldsymbol{x}) \right\rVert_2 \le L_\boldsymbol{x} \left\lVert \boldsymbol{r} \right\rVert_2.\]
6.1 First-Order Expansion and Residual Bound↩︎
By the fundamental theorem of calculus, the change in loss can be written exactly as an integral of the gradient: \[\ell(\boldsymbol{x}+\boldsymbol{r}) - \ell(\boldsymbol{x}) = \int_0^1 \bigl\langle \nabla_{\boldsymbol{x}} \ell(\boldsymbol{x}+t\boldsymbol{r}), \boldsymbol{r} \bigr\rangle \mathrm{d}t\] Inserting \(\nabla_{\boldsymbol{x}} \ell(\boldsymbol{x})\) inside the integral, we decompose the difference into a first-order linear term and a remainder \(R_2(\boldsymbol{x},\boldsymbol{r})\): \[\begin{align} \ell(\boldsymbol{x}+\boldsymbol{r}) - \ell(\boldsymbol{x}) &= \bigl\langle \nabla_{\boldsymbol{x}} \ell(\boldsymbol{x}), \boldsymbol{r} \bigr\rangle \\ &\quad + \underbrace{\int_0^1 \bigl\langle \nabla_{\boldsymbol{x}} \ell(\boldsymbol{x}+t\boldsymbol{r}) - \nabla_{\boldsymbol{x}} \ell(\boldsymbol{x}), \boldsymbol{r} \bigr\rangle \mathrm{d}t}_{R_2(\boldsymbol{x},\boldsymbol{r})}. \end{align}\] Taking absolute value and applying the Cauchy–Schwarz inequality together with the Lipschitz condition: \[\begin{align} |R_2(\boldsymbol{x},\boldsymbol{r})| &\le \int_0^1 \left\lVert \nabla_{\boldsymbol{x}} \ell(\boldsymbol{x}+t\boldsymbol{r}) - \nabla_{\boldsymbol{x}} \ell(\boldsymbol{x}) \right\rVert_2 \left\lVert \boldsymbol{r} \right\rVert_2 \mathrm{d}t \\ &\le \int_0^1 L_{\boldsymbol{x}} t \left\lVert \boldsymbol{r} \right\rVert_2^2 \mathrm{d}t = \frac{L_{\boldsymbol{x}}}{2} \left\lVert \boldsymbol{r} \right\rVert_2^2. \end{align}\] Since \(\|\boldsymbol{r}\| \le \rho\), the residual is bounded by \(\frac{L_{\boldsymbol{x}}}{2}\rho^2\).
6.2 Local Adversarial Sensitivity↩︎
The local adversarial sensitivity is defined as the maximal loss increase within the \(\rho\)-ball: \[\begin{align} S_{\rho}(\boldsymbol{x},y;\theta) := \max_{\left\lVert \boldsymbol{r} \right\rVert_2\le\rho} [\ell(\boldsymbol{x}+\boldsymbol{r})-\ell(\boldsymbol{x})]\\ = \max_{\left\lVert \boldsymbol{r} \right\rVert_2\le\rho} \big[\langle \nabla_x \ell(\boldsymbol{x}), \boldsymbol{r} \rangle + R_2(\boldsymbol{x},\boldsymbol{r})\big]. \end{align}\] By Cauchy–Schwarz, the first-order term is maximized when \(\boldsymbol{r}\) is aligned with \(\nabla_{\boldsymbol{x}} \ell(\boldsymbol{x})\), giving \(\rho\left\lVert \nabla_\boldsymbol{x} \ell(\boldsymbol{x}) \right\rVert_2\). Combined with the residual bound, the maximum absolute error is \(O(\rho^2)\). Hence the first-order approximation \[S_{\rho}(\boldsymbol{x},y;\theta) = \rho \left\lVert \nabla_\boldsymbol{x} \ell(\boldsymbol{x}) \right\rVert_2 + O(\rho^2)\] holds rigorously.
7 First-Order Spectral Analysis of Effective Rank↩︎
We derive the first-order Taylor expansion of the effective rank of the perturbed empirical NTK. Let \(\boldsymbol{\Theta}\in\mathbb{R}^{N\times N}\) be a nonzero symmetric positive-semidefinite matrix. The effective rank is \[r_{\mathrm{eff}}(\boldsymbol{\Theta}) = \frac{\mathop{\mathrm{tr}}(\boldsymbol{\Theta})^2}{\left\lVert \boldsymbol{\Theta} \right\rVert_F^2}.\] Consider the perturbed empirical NTK \[\widetilde{\boldsymbol{\Theta}}(\rho) = \boldsymbol{\Theta} + \rho \boldsymbol{E} + \boldsymbol{R}(\rho), \qquad \left\lVert \boldsymbol{R}(\rho) \right\rVert_F = o(\rho),\] where \(\boldsymbol{E}\) is symmetric and \(\widetilde{\boldsymbol{\Theta}}(\rho)\succeq 0\) for sufficiently small \(\rho>0\). The Frobenius inner product is \(\langle \boldsymbol{A},\boldsymbol{B}\rangle = \mathop{\mathrm{tr}}(\boldsymbol{A}^\top \boldsymbol{B})\).
Lemma 2. Let \(T=\operatorname{tr}(\boldsymbol{\Theta})>0\) and \(S=\|\boldsymbol{\Theta}\|_{\mathrm{F}}^2>0\). The perturbed effective rank satisfies \[r_{\mathrm{eff}}(\widetilde{\boldsymbol{\Theta}}(\rho)) = r_{\mathrm{eff}}(\boldsymbol{\Theta}) + 2\,r_{\mathrm{eff}}(\boldsymbol{\Theta})\left( \frac{\operatorname{tr}(\boldsymbol{E})}{\operatorname{tr}(\boldsymbol{\Theta})} - \frac{\langle\boldsymbol{\Theta},\boldsymbol{E}\rangle}{\|\boldsymbol{\Theta}\|_{\mathrm{F}}^2} \right)\rho + o(\rho).\]
Proof. Using linearity of the trace, \[\operatorname{tr}(\widetilde{\boldsymbol{\Theta}}(\rho)) = T + \rho\operatorname{tr}(\boldsymbol{E}) + o(\rho).\] For the denominator, \[\|\widetilde{\boldsymbol{\Theta}}(\rho)\|_{\mathrm{F}}^2 = \|\boldsymbol{\Theta} + \rho \boldsymbol{E} + \boldsymbol{R}(\rho)\|_{\mathrm{F}}^2 = S + 2\rho\langle\boldsymbol{\Theta},\boldsymbol{E}\rangle + o(\rho),\] where cross terms involving \(\boldsymbol{R}(\rho)\) are \(o(\rho)\). Substituting into the effective rank: \[r_{\mathrm{eff}}(\widetilde{\boldsymbol{\Theta}}(\rho)) = \frac{\bigl(T+\rho\operatorname{tr}(\boldsymbol{E})+o(\rho)\bigr)^2}{S+2\rho\langle\boldsymbol{\Theta},\boldsymbol{E}\rangle+o(\rho)} = \frac{T^2 + 2\rho T\operatorname{tr}(\boldsymbol{E}) + o(\rho)}{S + 2\rho\langle\boldsymbol{\Theta},\boldsymbol{E}\rangle + o(\rho)}.\] Apply the first-order expansion \[\frac{A+\rho a}{B+\rho b} = \frac{A}{B}\left(1+\rho\frac{a}{A}-\rho\frac{b}{B}\right)+o(\rho)\] with \(A=T^2\), \(a=2T\operatorname{tr}(\boldsymbol{E})\), \(B=S\), \(b=2\langle\boldsymbol{\Theta},\boldsymbol{E}\rangle\). Recalling \(r_{\mathrm{eff}}(\boldsymbol{\Theta})=T^2/S\), \[r_{\mathrm{eff}}(\widetilde{\boldsymbol{\Theta}}(\rho)) = r_{\mathrm{eff}}(\boldsymbol{\Theta}) \left(1+\rho\frac{2T\operatorname{tr}(\boldsymbol{E})}{T^2}-\rho\frac{2\langle\boldsymbol{\Theta},\boldsymbol{E}\rangle}{S}\right)+o(\rho).\] Simplifying yields the statement of the lemma. ◻
8 Evaluation Metrics↩︎
This appendix defines the metrics used in the experiments and states the preferred direction for each metric.
8.0.0.1 Selected Feature Count and Sparsity.
For a selected feature set \(A\) and input dimension \(d\), \[\text{Sparsity}=\frac{|A|}{d}.\] Lower sparsity indicates a smaller selected support, provided that predictive performance is maintained.
8.0.0.2 Jaccard Index.
For two selected feature sets \(A\) and \(B\) from independent runs, \[J(A,B)=\frac{|A\cap B|}{|A\cup B|}.\] Higher values indicate more reproducible feature selection.
8.0.0.3 ID and OOD Accuracy.
Classification accuracy is \[\text{Accuracy}=\frac{\text{Number of correct predictions}}{\text{Total number of samples}}.\] Higher values indicate better predictive performance. ID accuracy is measured on in-distribution data, whereas OOD accuracy is measured under the specified distribution shift.
8.0.0.4 ID–OOD Gap.
We quantify robustness under shift as \[\text{Gap}=\text{ID Accuracy}-\text{OOD Accuracy}.\] Lower values indicate a smaller degradation from ID to OOD evaluation.
8.0.0.5 Empirical NTK Effective Rank.
For an empirical NTK matrix \(\Theta\in\mathbb{R}^{N\times N}\), \[r_{\mathrm{eff}}(\Theta)=\frac{\operatorname{tr}(\Theta)^2}{\|\Theta\|_F^2}.\] Higher values indicate a less concentrated NTK spectrum.
8.0.0.6 Leading Eigenvalue Ratio.
Let \(\lambda_1(\Theta)\) be the largest eigenvalue of \(\Theta\). We report \[\frac{\lambda_1(\Theta)}{\operatorname{tr}(\Theta)}.\] Lower values indicate that the dominant spectral mode accounts for a smaller fraction of total kernel energy.
8.0.0.7 Top-\(k\) Energy Ratio.
For eigenvalues \(\lambda_1(\Theta)\ge \cdots \ge \lambda_N(\Theta)\ge 0\), \[\frac{\sum_{i=1}^{k}\lambda_i(\Theta)}{\operatorname{tr}(\Theta)}\] measures the fraction of spectral energy in the leading \(k\) directions. We use \(k=10\). Lower values indicate less concentration in the leading eigendirections.
8.0.0.8 Hessian Diagnostics.
Let \(H\) be the Hessian of the training objective with respect to model parameters. We report the maximum eigenvalue \(\lambda_{\max}(H)\), the trace \(\operatorname{tr}(H)\), and the average curvature \[\bar{\kappa}=\frac{\operatorname{tr}(H)}{\dim(H)}.\] Lower values indicate milder local curvature.
9 Supplementary Analysis on Controlled Feature Budgets↩︎
To complement the SERS evaluation in the main text, we report controlled feature-budget results under two sparsity regimes. This analysis separates the effect of adversarial stability from the effect of selecting more features.
| Budget / Model | Val Acc | Test Acc | Sens | Spec | AUC | Features |
|---|---|---|---|---|---|---|
| High-Sparsity Regime (\(k \approx 120\)) | ||||||
| Standard (clean) | 75.71 | 72.14 | 85.71 | 58.57 | 0.775 | 119 |
| Adversarial (sam) | 47.86 | 52.86 | 62.86 | 42.86 | 0.482 | 122 |
| \(\Delta\) (sam \(-\) clean) | \(-27.86\) | \(-19.29\) | \(-22.86\) | \(-15.71\) | \(-0.293\) | \(+3\) |
| Moderate-Sparsity Regime (\(k \approx 250\)) | ||||||
| Standard (clean) | 57.14 | 62.14 | 85.71 | 38.57 | 0.672 | 272 |
| Adversarial (sam) | 73.57 | 76.43 | 71.43 | 81.43 | 0.825 | 256 |
| \(\Delta\) (sam \(-\) clean) | \(+16.43\) | \(+14.29\) | \(-14.29\) | \(+42.86\) | \(+0.154\) | \(-16\) |
Under the high-sparsity budget, the adversarial model selects nearly the same number of features as the standard model but performs worse. Under the moderate-sparsity budget, it improves test accuracy and AUC while selecting fewer features. These results suggest that the SERS gains in the main experiments are not explained by feature-count expansion alone. Instead, the effect of adversarial stability depends on whether the feature budget allows a sufficiently informative and stable support to be retained.