1 Introduction↩︎

Perineural invasion (PNI) refers to the infiltration of tumour cells into the space surrounding peripheral nerves, constituting a distinct pathway for tumour spread beyond the primary site [1]. In cholangiocarcinoma (CCA), PNI has been closely linked to higher recurrence rates, lymph node metastasis, and diminished overall survival, establishing it as a prognostic factor following surgical intervention [2]–[4]. Preoperative identification of PNI is clinically valuable for guiding the extent of surgical resection [5] and neoadjuvant treatment decisions [6]; however, PNI manifests as microscopic neural tracking at the tumour periphery [1], making non-invasive detection inherently difficult [7]–[9]. Conventional quantitative approaches have predominantly employed radiomics-based analysis [10], which demands extensive manual delineation of tumour boundaries [11], [12].

Recent advances in 3D deep learning have enabled automated volumetric analysis for medical imaging. CNN-based architectures such as ResNet [13], DenseNet [14], and EfficientNet [15] have been widely adopted for 3D classification tasks, while Transformer-based models like Swin Transformer [16] leverage window-based self-attention to capture long-range dependencies. However, applying self-attention to high-resolution 3D volumes incurs a computational burden that scales quadratically with the number of tokens, making it impractical for resource-constrained clinical settings.

Spiking Neural Networks (SNNs) offer a fundamentally different computational paradigm that addresses this energy bottleneck. Unlike conventional Artificial Neural Networks (ANNs) that rely on energy-intensive floating-point multiply-accumulate (MAC) operations, SNNs communicate via asynchronous binary spikes; when activations are zero, the corresponding multiplications can be entirely skipped, reducing both operations and energy [17], [18]. Spikformer [19] first demonstrated that spiking self-attention (SSA), using spike-form queries, keys, and values without softmax, can achieve competitive performance on vision tasks. The Spike-driven Transformer (SD-Transformer) [20] further advanced this line by proposing a purely spike-driven attention mechanism, and its successor Meta-SpikeFormer [21] established a versatile meta-architecture for SNNs. More recently, the Spiking Transformer with Experts Mixture (SEMM) [22] introduced a binary routing mechanism that selectively activates expert heads through Hadamard masking, realising sparse, event-driven mixture-of-experts routing entirely within the spike domain.

Despite these advances, existing spiking transformers process all spatial tokens uniformly [23], overlooking the inherent spatial redundancy in 3D medical volumes where large background regions carry negligible diagnostic information [24]. We introduce SpikeDS, a Dual Sparsity Spikformer that combines the activation sparsity inherent in SNNs with an explicit spatial sparsity mechanism. SpikeDS partitions the 3D volume into non-overlapping windows and prunes uninformative ones based on firing rates, concentrating attention computation on diagnostically relevant regions. A complementary cross-window attention module then maintains global information exchange between the retained and pruned windows through an asymmetric query–key scheme. This dual sparsity is applied progressively across encoder stages, allowing early layers to build reliable low-level features before deeper layers focus computation on the most informative regions.

2 Method↩︎

2.1 Overall Architecture↩︎

Fig 1 illustrates the overall architecture of our proposed method. A 3D input volume \(\mathbf{X}\!\in\!\mathbb{R}^{C\times H\times W\times Z}\) is first encoded into spiking feature maps by a Spike Patch Splitting (SPS) module. The tokens are then processed by a three-stage hierarchical encoder with increasing channel dimensions and decreasing spatial resolutions, connected via 3D patch merging layers. Each stage stacks multiple SpikeDS Blocks that comprise a sequence of DSSA \(\to\) CW-SSA \(\to\) Spiking MLP, with residual connections throughout. In Stage 0, all windows are processed without pruning; from Stage 1 and 2 onward, firing-rate-guided window pruning is activated. Global average pooling, batch normalization, a LIF layer, and temporal mean reduction yield the final representation for classification.

2.2 Spike Patch Splitting (SPS)↩︎

SPS  [19] converts the continuous-valued input into discrete spike representations through four cascaded Conv3D\(\,\to\,\)BN\(\,\to\,\)LIF stages. Spatial down-sampling via max pooling is applied selectively in the first two stages, reducing the resolution to one-quarter of the original. A convolutional relative position encoding (RPE) branch injects local spatial priors before the tokens enter the transformer encoder. We employ the Leaky Integrate-and-Fire (LIF) neuron model [25] with hard reset, where the membrane potential \(U[t]\) accumulates input features and fires a binary spike \(S[t]\) upon exceeding a threshold \(V_\text{th}\): \[U[t]=\alpha\!\cdot\!U[t\!-\!1]\!\cdot\!(1\!-\!S[t\!-\!1])+X[t],\; S[t]=\Theta(U[t]-V_\text{th}) \label{eq:lif}\tag{1}\] where \(\alpha=1-1/\tau\) is the membrane decay factor and \(\Theta(\cdot)\) is the Heaviside step function. Since the spike outputs are binary, downstream layers incur only accumulate (AC) operations rather than energy-intensive MAC operations. Crucially, the first Conv3D layer of SPS receives continuous-valued inputs and therefore requires MAC operations; all subsequent layers operate in the spike domain and incur only AC operations.

2.3 Dual Sparsity Spiking Attention (DSSA)↩︎

DSSA unifies firing-rate-guided window pruning and window-based expert mixture spiking attention (W-EMSA) into a single module. Given input \(\mathbf{X}\!\in\!\mathbb{R}^{T\times B\times H\times W\times Z\times D}\), where \(\mathbf{B}\) is the batch size, we partition it into \(N_w\) non-overlapping 3D windows \(\{W_j\}_{j=1}^{N_w}\) of size \(w_H\!\times\!w_W\!\times\!w_Z\). A saliency score for each window is computed as its mean firing rate: \[s_j = \frac{1}{TBMD}\sum_{t,b,m,d} W_j^{(t,b,m,d)} \label{eq:saliency}\tag{2}\] where \(M\!=\!w_H w_W w_Z\) is the token count per window. The active set \(\mathcal{A}=\text{top-}k(\{s_j\},\,\lfloor\rho N_w\rfloor)\) is selected, where \(\rho\!\in\!(0,1]\) denotes the proportion of windows retained after pruning, and DSSA is defined as: \[\text{DSSA}(W_j) = \begin{cases} \text{W-EMSA}(W_j), & j\in\mathcal{A}\\ W_j, & j\notin\mathcal{A} \end{cases} \label{eq:dssa}\tag{3}\] In Stage 0, pruning is disabled (\(\mathcal{A}=\{1,\ldots,N_w\}\)).

W-EMSA performs window-based expert mixture spiking attention inspired by SEMM [22]. Tokens pass through a LIF layer and are projected into queries, keys, and values via spiking linear layers (Linear\(\,\to\,\)BN\(\,\to\,\)LIF). Since \(\mathbf{Q}\), \(\mathbf{K}\), \(\mathbf{V}\) are all binary spike tensors after the LIF layers, the matrix products degenerate to sparse accumulate operations, yielding energy-efficient attention. Each attention head is treated as a specialised expert: \[\text{SSA}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \text{SN}\!\bigl( \mathbf{Q}\mathbf{K}^\top\mathbf{V}\cdot s\bigr),\;s = d^{-1/2} \label{eq:attn}\tag{4}\]

\[A_j^{(h)} = \text{SSA}^{(h)}(\mathbf{Q}_j^{(h)},\, \mathbf{K}_j^{(h)},\,\mathbf{V}_j^{(h)}) \label{eq:expert}\tag{5}\] A spiking router generates a binary gating vector from the window-level mean token \(\bar{w}_j\): \[\mathbf{r}_j = \text{SN}\!\bigl(\text{BN}(\bar{w}_j\, \mathbf{W}_R)\bigr) \in \{0,1\}^{H} \label{eq:gate}\tag{6}\] The gated head outputs are concatenated and projected through a final spiking linear layer: \[\mathbf{O}_j = \bigl[A_j^{(1)};\,\dots\,;\,A_j^{(H)}\bigr] \in\{0,1\}^{M\times D} \label{eq:head95concat}\tag{7}\] \[\text{W-EMSA}_j = \text{SN}\!\Bigl(\text{BN}\!\bigl( (\mathbf{r}_j \odot \mathbf{O}_j)\,\mathbf{W}_O\bigr)\Bigr) \label{eq:router}\tag{8}\] Since the router outputs binary spikes, each \(r_j^{(h)} \in \{0,1\}\) acts as a binary mask that selectively activates or suppresses the corresponding head expert per window, preserving the spike-driven property of the network.

2.4 Cross-Window Spiking Self-Attention (CW-SSA)↩︎

CW-SSA enables global information exchange across all windows. It is important to note that the window pruning introduced in DSSA excludes low-saliency windows only from the W-EMSA attention computation; the windows themselves and their feature representations are not discarded. CW-SSA exploits this by allowing pruned windows to remain accessible as key-value sources for global context aggregation.

Concretely, each window is mean-pooled to a representative token \(\bar{\mathbf{w}}_j\). When pruning is active, CW-SSA adopts an asymmetric query–key/value scheme: \[\mathbf{Q}\!=\!f_q(\{\bar{\mathbf{w}}_j\}_{j\in\mathcal{A}}),\; \mathbf{K}\!=\!f_k(\{\bar{\mathbf{w}}_j\}_{j=1}^{N_w}),\; \mathbf{V}\!=\!f_v(\{\bar{\mathbf{w}}_j\}_{j=1}^{N_w}) \label{eq:cwssa}\tag{9}\] Queries are derived only from the \(\rho N_w\) active windows, while keys and values are drawn from all \(N_w\) windows, including those pruned from W-EMSA. This design is motivated by three considerations. First, pruned windows retain their stage-input representations from the preceding patch merging layer (or SPS for Stage 0), which, although not refined by W-EMSA, still encode meaningful low-level spatial features. Second, the overhead of projecting all \(N_w\) tokens into keys and values is negligible because \(N_w\) is the window-level token count, not the voxel-level count. Third, the resulting attention map has shape \(\rho N_w \times N_w\), allowing active windows to attend to the full spatial extent of the volume while the \(\rho\) factor still reduces query-side computation. The cross-attended tokens are broadcast back to active windows as a residual. In Stage 0 (no pruning), CW-SSA operates symmetrically with \(\rho\!=\!1\).

3.5pt

2.5 SpikeDSBlock and Hierarchical Encoder↩︎

Each SpikeDSBlock follows: \[\hat{\mathbf{X}} = \mathbf{X} + \text{DSSA}(\mathbf{X}) + \text{CW-SSA}(\text{DSSA}(\mathbf{X})),\;\; \label{eq:block}\tag{10}\] \[\mathbf{X}' = \hat{\mathbf{X}} + \text{SpikingMLP}(\hat{\mathbf{X}})\]

The three-stage hierarchical encoder doubles the channel dimension and halves the spatial resolution at each transition via a 3D patch merging layer. Window pruning is disabled in the first stage to allow reliable low-level feature formation, and is activated with keep ratio \(\rho\) in the deeper stages.

3 Experiments↩︎

3.1 Dataset and Implementation Details↩︎

Dataset. We utilised a retrospective clinical cohort of 139 CCA patients (PNI-positive: 85, PNI-negative: 54) with pathologically confirmed PNI status, collected over more than a decade at Samsung Medical Center. The T2-weighted phase of contrast-enhanced MRI was used as the single-phase input. Each volume was cropped to a fixed \(96\!\times\!96\!\times\!48\) region centred on the tumor [26]. A three-channel input was constructed by stacking the normalized intensity image, a liver mask, and a tumor mask.

Implementation. SpikeDS was implemented in PyTorch with SpikingJelly [27] and evaluated using stratified 5-fold cross-validation. The model was optimized with AdamW [28] (learning rate \(1\!\times\!10^{-4}\), weight decay \(1\!\times\!10^{-5}\)) for 100 epochs with a batch size of 2 using mixed-precision training on a single NVIDIA A100 GPU. A cosine annealing schedule was applied. The base channel dimension was set to \(C\!=\!64\) and the number of LIF timesteps to \(T\!=\!4\).

3.2 Quantitative Evaluation and Visualization↩︎

We benchmarked SpikeDS against CNN and SNN baselines using the identical three-channel input (intensity, liver mask, tumor mask). As shown in Table 1, 3D CNN models achieved AUCs between 0.621 and 0.711, while SNN baselines attained comparable AUCs at substantially lower energy consumption. SpikeDS achieves the highest AUC of 0.753 at only 14.4 mJ, the lowest among all SNN baselines. Although EfficientNet-B0 consumes less energy (2.76 mJ), its AUC of 0.621 is the lowest in the comparison, whereas SpikeDS maintains the best AUC–energy trade-off across all models.

Fig. 2 visualizes the spiking firing rate (SFR) maps on representative axial slices. SpikeDS concentrates higher firing rates around the tumor region, indicating that the dual sparsity mechanism effectively directs computational focus toward diagnostically relevant areas.

3.3 Ablation Study↩︎

Table 2 summarizes the ablation results. Removing window pruning raises energy to 16.17 mJ and lowers AUC to 0.674, as indiscriminate full-window attention introduces background noise. Disabling CW-SSA reduces AUC to 0.732. Replacing its asymmetric design with a symmetric variant that excludes pruned windows from the key–value pool further confirms the value of pruned-window context: the asymmetric design achieves 0.753 AUC, while the symmetric variant drops to 0.737. Replacing expert routing with standard W-SSA drops AUC to 0.729 at 16.16 mJ, losing head-wise suppression of uninformative attention. Pruning at all stages degrades AUC to 0.675 because early features are insufficiently formed for reliable saliency estimation, while pruning only at Stage 2 yields suboptimal efficiency of 14.47 mJ. The adopted configuration—pruning with \(\rho\!=\!0.5\) at Stages 1 and 2 only—achieves the best trade-off, achieving AUC of 0.753 at 14.41 mJ.

4pt

3.4 Energy Analysis↩︎

SpikeDS combines activation sparsity (binary spikes) and spatial sparsity (window pruning). All energy estimates adopt the 45 nm model [29] with \(E_\text{MAC}=4.6\) pJ and \(E_\text{AC}=0.9\) pJ. Binary spike activations reduce MACs to ACs in all layers except the first (float-input) Conv3D, following the standard SOP formulation [19], [20]: \[\text{SOP}(l) = \bar{s}_l \cdot T \cdot \text{FLOPs}(l) \label{eq:sop}\tag{11}\] Each SpikeDS Block incurs energy modulated by the keep ratio \(\rho\): \[E_\text{block} = E_\text{AC}\cdot\bigl(\rho\,\Phi_\text{E} + \Phi_\text{C} + \Phi_\text{M}\bigr) \label{eq:block95energy}\tag{12}\] where \(\Phi_\text{E}\), \(\Phi_\text{C}\), and \(\Phi_\text{M}\) denote the SOP counts of W-EMSA, CW-SSA, and Spiking MLP, respectively. When \(\rho\!=\!1\) (Stage 0) the block reduces to a standard spiking transformer; when \(\rho\!<\!1\) (Stages 1–2), W-EMSA cost decreases proportionally.

Total inference energy is: \[\begin{gather} E_\text{total} = E_\text{MAC}\cdot\text{FL}_1 \\ + E_\text{AC}\cdot\Bigl(\text{SOP}_\text{SPS} + \sum_{i}\bigl[d_i\cdot\tfrac{E_\text{block}^{(i)}}{E_\text{AC}} + \text{SOP}_\text{PM}^{(i)}\bigr] + \text{SOP}_\text{head}\Bigr) \label{eq:total95energy} \end{gather}\tag{13}\] where \(d_i\) is the block depth of stage \(i\) (\(d=2,2,4\)).

Activation sparsity alone (\(\bar{s}\approx0.10\)) yields \({\sim}82\%\) energy saving over the ANN-equivalent configuration of the same architecture (92.3 mJ \(\to\) 16.17 mJ). Window pruning at Stages 1–2 (\(\rho=0.5\)) compounds this further for a total saving of 84% (92.3 mJ \(\to\) 14.41 mJ).

4 Conclusion↩︎

We introduced SpikeDS, a spiking transformer that exploits activation sparsity and spatial sparsity for energy-efficient PNI prediction in 3D MRI. By introducing DSSA, which unifies firing-rate window pruning with W-EMSA, and CW-SSA, which preserves global context even for pruned regions through asymmetric cross-window attention, SpikeDS achieves a principled balance between computational efficiency and classification performance. The progressive pruning strategy ensures that spatial sparsity is applied only where saliency estimation is reliable. We provided a decomposed energy analysis that explicitly separates the contributions of activation sparsity and spatial sparsity, demonstrating that their compounding effect yields substantial savings. Evaluated on a clinical CCA cohort for PNI prediction, SpikeDS demonstrates the potential of dual-sparsity spiking architectures for volumetric medical imaging under stringent energy constraints.

Acknowledgment↩︎

This work was supported by the IITP grant (IITP-2023-RS-2023-00256081) funded by MSIT, Korea, and the ANCHOR program (2026-ANCHOR-01-110) funded by the Ministry of Education and the Seoul Metropolitan Government, Republic of Korea.

References↩︎