1 Introduction↩︎

Rain in outdoor videos introduces streak-like occlusions, contrast attenuation, background distortion, and temporal flickering [1]. These degradations not only reduce visual quality, but also impair downstream vision tasks such as object detection, tracking, semantic segmentation, and autonomous driving [2]–[4]. Video deraining therefore plays an important role in reliable visual perception under adverse weather.

Existing deraining methods have been investigated in both single-image and video settings. Single-image deraining relies mainly on intra-frame spatial priors or learned representations to recover a clean image from a rainy observation [5]–[7]. However, without temporal information, rain streaks, fine textures, object boundaries, and motion-induced structures may exhibit similar local appearances, making rain-background separation ill-posed. Video deraining alleviates this ambiguity by exploiting temporal redundancy and inter-frame complementarity [8], [9]. Early methods typically model rainy videos as the superposition of latent clean backgrounds and rain layers, and introduce handcrafted priors such as temporal coherence, chromatic consistency, low-rank background structure, directional sparsity, and stochastic rain statistics [10]–[12]. Although interpretable, these assumptions are often too restrictive for dynamic videos with heavy rain, camera motion, occlusions, and spatially non-uniform degradation.

Recent deep video deraining methods replace handcrafted modeling with data-driven spatio-temporal restoration. They improve rain removal through motion alignment, recurrent refinement, progressive reconstruction, flow regularization, self-aligned temporal aggregation, and long-range temporal modeling [9], [13]–[15]. Some methods further consider complex rain effects such as rain accumulation, transmittance variation, and occlusion-related degradation [13], [16]. Despite these advances, RGB-only video deraining remains challenging in dynamic rainy scenes. Rain streaks, background textures, object boundaries, camera motion, and occlusions may share similar spatial patterns or temporal variations. As a result, a model may mistakenly suppress true structures as rain or preserve rain artifacts as background details, especially under heavy rain or fast motion. This limitation suggests that temporal redundancy from RGB frames alone may be insufficient, and complementary sensing cues are desirable for more reliable rain-background discrimination.

Event cameras provide such a complementary modality. Unlike frame-based cameras, event cameras asynchronously record brightness changes at each pixel with high temporal resolution [17], [18]. Each event is represented by its pixel location, timestamp, and polarity, and is triggered when the log-intensity variation exceeds a contrast threshold. This sensing mechanism produces sparse, low-latency, and motion-sensitive measurements, which have been widely used in motion-centric and restoration tasks, including optical flow estimation, camera tracking, event-to-frame reconstruction, motion deblurring, frame interpolation, super-resolution, and low-light enhancement [19]–[22]. These studies show that events can provide fine-grained temporal cues that are difficult to capture with conventional frame cameras.

The properties of event cameras are particularly relevant to video deraining. Rain streaks often induce rapid, localized, and non-uniform brightness changes, which may produce informative event responses. As illustrated in Fig. 1, rain streaks that are visually entangled with background textures in RGB frames can become more distinguishable in the event domain. However, event streams are not clean rain indicators. They may also contain sensor noise, background-triggered events, and responses caused by camera or object motion. Therefore, the key challenge is not simply to inject events into RGB restoration, but to selectively exploit rain-related event cues while suppressing cross-modal interference.

Recent event-guided video deraining methods have begun to explore RGB–Event collaboration [23], [24]. MPEVNet introduces event features into a multi-stage RGB–Event fusion network and can be regarded as an early-fusion strategy [25]. EGVD adopts a late-fusion scheme, where modality-specific restoration is first performed and the restored representations are fused afterwards [24]. Other event-aware methods further exploit event representations for rain modeling or heterogeneous feature interaction [23], [26]. These methods demonstrate the potential of event guidance, but their fusion mechanisms still operate mainly on mixed modality features. Early fusion may entangle RGB appearance, rain-sensitive event responses, and background-triggered event noise before rain and background factors are clarified. Late fusion, on the other hand, may weaken transient event cues before they can effectively guide RGB restoration. Consequently, existing fusion paradigms lack an explicit mechanism for aligning and interacting rain-related and background-related information across modalities. As shown in Fig. 2, residual rain artifacts and background structure loss remain visible in representative RGB–Event deraining results.

Spiking neural networks (SNNs) provide a natural computational model for event-based processing because they encode information through discrete spikes and temporal neuronal dynamics [27]. With surrogate-gradient training, residual structures, normalization, attention mechanisms, and spike-based Transformer architectures, SNNs have been extended from recognition to event-based perception and restoration tasks [28]–[30]. In adverse-weather imaging, SNNs have also been explored for event-driven deraining and rain-background decomposition [31], [32]. These studies indicate that spiking dynamics are useful for modeling sparse and transient event responses. Nevertheless, most existing SNN-based deraining methods focus on event representation or direct RGB–Event fusion, without explicitly distinguishing the semantic roles of event responses. For event-guided video deraining, this distinction is important because rain-induced events and background-triggered events often coexist in the same event stream.

To address these issues, we propose a decompose-before-interact paradigm for event-guided video deraining. Instead of directly fusing heterogeneous RGB and event features, we first disentangle rain and background components within each modality, and then perform cross-modal interaction between semantically corresponding components. In this way, rain-related event responses can guide RGB rain removal, while background-related representations are fused separately to preserve scene structures. This design reduces the interference between rain cues and background cues while retaining the complementary advantages of both modalities.

Based on this principle, we propose RainDancer, a progressive RGB–Event video deraining framework. At each restoration stage, the RGB branch decomposes frame features into background and rain representations. For the event branch, we design a rain-oriented SNN module to model sparse and bursty event dynamics associated with rain motion, thereby separating rain-related and background-related event responses more explicitly. The two modalities are then coupled through component-level interactions, including background-to-background fusion and rain-to-rain fusion. In addition, we introduce event-domain supervision to regularize sparse event reconstruction, structural consistency, and gradient orientation, improving the reliability of event representations used for RGB restoration. As shown in Fig. 2, the proposed framework removes rain more thoroughly while preserving background structures more faithfully than representative RGB–Event fusion baselines.

The main contributions of this work are summarized as follows:

• We propose a progressive RGB–Event video deraining framework (RainDancer) based on a decompose-before-interact principle. Different from conventional early or late fusion, RainDancer decomposes rain and background components before cross-modal interaction, thereby reducing premature feature entanglement and coarse representation aggregation.

• We design a component-level RGB–Event interaction mechanism that separately performs background-to-background and rain-to-rain fusion. By aligning semantically corresponding components across modalities, the proposed mechanism strengthens event-guided rain removal while preserving background structures.

• We develop a rain-oriented SNN event branch to capture sparse and bursty event dynamics induced by rain motion. Together with event-domain supervision on sparse reconstruction, structural consistency, and gradient orientation, the event branch provides more reliable event guidance for RGB video deraining.

2 Proposed Model↩︎

2.1 Overview↩︎

Fig. 3 illustrates the proposed progressive RGB–Event video deraining framework (RainDancer). Given a rainy RGB clip \(\mathbf{X}=[\mathbf{I}_{t-1}^{r},\mathbf{I}_{t}^{r},\mathbf{I}_{t+1}^{r}]\) and the synchronized event tensor \(\mathbf{E}\in\mathbb{R}^{20\times H\times W}\), the network predicts the restored RGB background \(\hat{\mathbf{B}}_{t}\), the clean event background \(\hat{\mathbf{E}}_{b}\), and the RGB rain component \(\hat{\mathbf{R}}_{t}\). These outputs explicitly correspond to the background and rain factors considered in our rain-background decomposition formulation.

The framework follows a decompose-before-interact principle. Instead of directly fusing RGB and event features, each modality first separates its representation into background and rain components. Cross-modal interaction is then performed between semantically aligned components, i.e., background-to-background fusion and rain-to-rain fusion. This design reduces the risk that rain-induced event responses or event noise are directly injected into RGB background restoration.

The RGB clip is first encoded by an Encoder block to obtain an initial spatio-temporal feature \(\mathbf{F}\). The following network contains three progressive stages with the same topology. At the \(s\)-th stage, the RGB branch decomposes the current RGB representation into background and rain features, while the event branch estimates event background and rain-motion features through a rain-oriented SNN event disentanglement module. The two pairs of features are then refined by coupled representation modules (CRMs). After intra-modal refinement, rain-level RGB–Event fusion and background-level RGB–Event fusion are performed by EGRF and RABF, respectively. The fused rain representation is stored for cross-stage aggregation, and the updated background representation is passed to the next stage.

The stage-wise process is formulated as \[\begin{align} (\mathbf{B}^{f}_{s},\mathbf{R}^{f}_{s}) &=\mathrm{CRM}_{f}^{s}\!\left(D_{f}^{s}(\mathbf{B}^{\mathrm{fus}}_{s-1})\right), \\ (\mathbf{B}^{e}_{s},\mathbf{R}^{e}_{s}) &=\mathrm{CRM}_{e}^{s}\!\left(D_{e}^{s}(\mathbf{E},\mathbf{R}^{e}_{s-1})\right), \\ \mathbf{R}^{\mathrm{fus}}_{s} &=\mathrm{EGRF}^{s}(\mathbf{R}^{f}_{s},\mathbf{R}^{e}_{s}), \\ \mathbf{B}^{\mathrm{fus}}_{s} &=\mathrm{RABF}^{s}(\mathbf{B}^{f}_{s},\mathbf{B}^{e}_{s}, \mathbf{R}^{f}_{s},\mathbf{R}^{e}_{s}), \end{align}\] where \(D_{f}^{s}(\cdot)\) and \(D_{e}^{s}(\cdot)\) denote frame-domain and event-domain decomposition, respectively. \(\mathbf{B}^{\mathrm{fus}}_{s}\) is the updated background feature, \(\mathbf{R}^{\mathrm{fus}}_{s}\) is the fused rain representation, and \(\mathbf{R}^{e}_{s-1}\) is the event-rain prior propagated from the previous stage. For the first stage, \(\mathbf{B}^{\mathrm{fus}}_{0}=\mathbf{F}\), and \(\mathbf{R}^{e}_{0}\) is initialized as zero.

After three stages, the rain features \(\{\mathbf{R}^{\mathrm{fus}}_{s}\}_{s=1}^{3}\) are aggregated into \(\mathbf{R}^{\mathrm{fus}}_{Agg}\). A recovery module then takes \(\mathbf{B}^{\mathrm{fus}}_{3}\), \(\mathbf{R}^{\mathrm{fus}}_{Agg}\), and \(\mathbf{B}^{e}_{3}\) as inputs and uses three lightweight decoders to predict \(\hat{\mathbf{B}}_{t}\), \(\hat{\mathbf{E}}_{b}\), and \(\hat{\mathbf{R}}_{t}\), respectively.

2.2 RGB Temporal Encoding and Frame-domain Decomposition↩︎

The RGB stream starts with a compact temporal encoder. A grouped 2D convolution first extracts frame-wise features from neighboring RGB frames while limiting premature cross-frame mixing. A lightweight 3D convolution then aggregates short-range temporal information and produces the initial feature \(\mathbf{F}\). This front-end provides the progressive stages with temporally aware RGB representations.

At each stage, the frame-domain decomposition block separates the current RGB feature into background and rain components. The background branch estimates stable scene structures using channel and spatial attention. Channel attention emphasizes feature channels associated with persistent appearance, while spatial attention identifies regions where background cues are more reliable. Let \(D_{f,b}^{s}(\cdot)\) denote the background estimation branch. The RGB decomposition is written as \[\mathbf{B}^{f}_{s}=D_{f,b}^{s}(\mathbf{B}^{\mathrm{fus}}_{s-1}), \qquad \mathbf{R}^{f}_{s}=\mathbf{B}^{\mathrm{fus}}_{s-1}-\mathbf{B}^{f}_{s}.\] The rain feature is obtained through a residual relation rather than an independent branch. This formulation is consistent with the additive rain-background decomposition assumption and constrains the two components to explain the input feature jointly.

Because rain and background may still be entangled after one-step decomposition, a CRM is used to refine the pair \(\left(\mathbf{B}^{f}_{s},\mathbf{R}^{f}_{s}\right)\). The CRM exchanges information between the two streams, allowing residual rain responses in the background stream to be transferred to the rain stream and misclassified background details to be compensated back. This refinement improves the reliability of subsequent component-level RGB–Event interaction.

2.3 Rain-oriented SNN Event Disentanglement↩︎

Event streams provide motion-sensitive cues for rain, but they also contain background-triggered responses and sensor noise. Directly using such mixed event features for RGB restoration can introduce cross-modal interference. We therefore design a rain-oriented SNN event disentanglement module, termed SNN-RoED, to first enhance rain-related event dynamics and then estimate event background features by residual decomposition.

Given the event tensor \(\mathbf{E}\), a base event branch extracts a compact event representation \(\mathbf{E}^{\mathrm{base}}_{s}\). In parallel, three rain-motion branches are used to capture complementary event cues. The local motion branch uses 3D convolutions and temporal pooling to aggregate short-term burst responses caused by fast rain streaks. The global spiking motion branch employs 3D convolutions followed by an SNNformer adapted from SpikeVideoFormer [33], which models long-range temporal dependencies in sparse event streams. The directional branch aggregates pre-event and post-event responses and applies 2D convolutions to enhance direction-aware rain traces.

The three motion cues are fused by a rain-motion gated unit. The event-rain estimate from the previous stage is used as a progressive prior: \[\begin{align} \mathbf{M}^{e}_{s} &=\Phi_{\mathrm{gate}}^{s} \left( [\mathbf{M}^{\mathrm{loc}}_{s}, \mathbf{M}^{\mathrm{spk}}_{s}, \mathbf{M}^{\mathrm{dir}}_{s}], \mathbf{R}^{e}_{s-1} \right), \\ \mathbf{R}^{e}_{s} &=\Phi_{r}^{s}(\mathbf{M}^{e}_{s}), \\ \mathbf{B}^{e}_{s} &=\Phi_{b}^{s}(\mathbf{E}^{\mathrm{base}}_{s}-\mathbf{R}^{e}_{s}). \end{align}\] Here, \(\mathbf{M}^{\mathrm{loc}}_{s}\), \(\mathbf{M}^{\mathrm{spk}}_{s}\), and \(\mathbf{M}^{\mathrm{dir}}_{s}\) denote local, spiking, and directional rain-motion cues, respectively. The residual form encourages the event background to be estimated after removing rain-dominant responses from the base event representation. The resulting pair \(\left(\mathbf{B}^{e}_{s},\mathbf{R}^{e}_{s}\right)\) is further refined by an event-domain CRM before cross-modal fusion.

2.4 Rain-Aware Background Fusion↩︎

The background fusion module, RABF, updates the RGB background feature using event background cues under rain-aware constraints. Its inputs are the RGB background feature \(\mathbf{B}^{f}_{s}\), the event background feature \(\mathbf{B}^{e}_{s}\), the RGB rain feature \(\mathbf{R}^{f}_{s}\), and the event rain feature \(\mathbf{R}^{e}_{s}\). The background components provide complementary structural information, while the rain components indicate regions where event responses are likely dominated by rain motion. Therefore, RABF uses rain features to regulate the reliability of event-guided background compensation.

RABF contains three branches. The interaction branch computes cross-modal relations from \([\mathbf{B}^{f}_{s},\mathbf{B}^{e}_{s}, \mathbf{B}^{f}_{s}\odot\mathbf{B}^{e}_{s}, |\mathbf{B}^{f}_{s}-\mathbf{B}^{e}_{s}|],\) where the product term measures feature consistency and the absolute difference highlights modality discrepancy. Convolutional layers then predict an event-guided background residual \(\Delta^{b}_{s}\). The gate branch takes \([\mathbf{B}^{f}_{s},\mathbf{B}^{e}_{s},\mathbf{R}^{f}_{s},\mathbf{R}^{e}_{s}]\) as input and predicts a rain-aware gate \(\mathbf{G}^{b}_{s}\). The confidence branch estimates an event confidence map \(\mathbf{C}^{b}_{s}\) from \([\mathbf{B}^{e}_{s},\mathbf{R}^{e}_{s}]\) to suppress unreliable event background cues.

The background feature is updated by a gated residual rule: \[\mathbf{B}^{\mathrm{fus}}_{s}= \Phi_{b,o}^{s}\!\left( \mathbf{B}^{f}_{s} + \beta_{b}\, \mathbf{G}^{b}_{s}\odot \mathbf{C}^{b}_{s}\odot \Delta^{b}_{s} \right),\] where \(\beta_b\) is a learnable residual scale. This conservative update injects event information only when it is both rain-aware and confidence-weighted, preventing noisy or rain-dominated event responses from corrupting the dense RGB background representation.

2.5 Event-guided Rain Fusion and Cross-stage Aggregation↩︎

The rain fusion module, EGRF, performs rain-to-rain RGB–Event interaction. RGB rain features encode appearance-level degradation, whereas event rain features provide sparse and motion-sensitive rain cues. EGRF aligns the two rain representations and predicts event-guided rain compensation without replacing the RGB rain stream.

Specifically, \(\mathbf{R}^{f}_{s}\) and \(\mathbf{R}^{e}_{s}\) are first projected to a common feature space and concatenated. Multi-scale depthwise convolution branches with kernel sizes \(3\), \(5\), and \(7\) are then used to capture rain streaks with different spatial extents. The module predicts an event-guided compensation term \(\Delta^{r}_{s}\) and a fused rain feature \(\mathbf{G}^{r}_{s}\). The final stage-wise rain representation is \[\mathbf{R}^{\mathrm{fus}}_{s} = \Phi_{r,o}^{s}\!\left( \mathbf{G}^{r}_{s} + \Delta^{r}_{s} + \Phi_{r,p}^{s}(\mathbf{R}^{f}_{s}) \right),\] where \(\Phi_{r,p}^{s}(\cdot)\) projects the RGB rain feature and \(\Phi_{r,o}^{s}(\cdot)\) generates the fused rain representation. This formulation anchors the fused rain feature to the RGB degradation representation while using event cues to enhance transient rain dynamics.

The stage-wise rain features are finally collected by the cross-stage aggregation module, which uses channel and spatial attention to adaptively combine complementary rain responses from all three stages. In this way, coarse rain structures captured in early stages and fine residual streaks refined in later stages are integrated into a more complete rain representation for final recovery.

2.6 Optimization Objectives↩︎

The model is optimized by an image-domain loss and an event-domain loss: \[\mathcal{L} = \mathcal{L}_{\mathrm{img}} + \mathcal{L}_{\mathrm{event}}.\]

The image loss supervises both RGB background restoration and rain-background decomposition. Following PCNet [34], we use Charbonnier loss, edge loss, and SSIM similarity for the restored background. An auxiliary residual reconstruction term is applied to the estimated rain component to regularize decomposition. The image-domain loss is defined as \[\begin{align} \mathcal{L}_{\mathrm{img}} ={}&0.3(\mathcal{L}_{\mathrm{char}}^{B} +0.2\mathcal{L}_{\mathrm{char}}^{I}) +0.2\mathcal{L}_{\mathrm{edge}}^{B} \\ &-0.15(\mathcal{L}_{\mathrm{ssim}}^{B} +0.2\mathcal{L}_{\mathrm{ssim}}^{I}), \end{align}\] where the residual reconstruction terms are assigned smaller weights because they serve as decomposition regularizers rather than the primary restoration objective.

The event-domain loss supervises the predicted clean event background \(\hat{\mathbf{E}}_{b}\). Since event tensors are sparse, a naive pixel-wise loss is dominated by inactive positions. Moreover, for event-guided deraining, the reliability of activity distribution, structural boundaries, and gradient orientation is more important than exact polarity matching. We therefore convert both predicted and ground-truth clean event tensors to their absolute values, denoted by \(\mathbf{E}_{p}\) and \(\mathbf{E}_{g}\), and define \[\mathcal{L}_{\mathrm{event}} = \lambda_{\mathrm{rec}}\mathcal{L}_{\mathrm{rec}} + \lambda_{\mathrm{grad}}\mathcal{L}_{\mathrm{grad}} + \lambda_{\mathrm{dir}}\mathcal{L}_{\mathrm{dir}}.\]

First, a sparsity-aware reconstruction loss emphasizes active event regions. Let\(\mathbf{A}=\sum_{c}\mathbf{E}_{g}^{c}\) be the event activity map over all bins. The activity-adaptive weight is \[\mathbf{W}=1+\alpha\left(\frac{\mathbf{A}}{\mathrm{mean}(\mathbf{A})+\epsilon}\right)^{\gamma}.\] The reconstruction term is \[\mathcal{L}_{\mathrm{rec}} = \frac{\sum \mathbf{W}\odot \rho(\mathbf{E}_{p}-\mathbf{E}_{g})}{\sum \mathbf{W}+\epsilon},\] where \(\rho(\cdot)\) is the Charbonnier penalty.

Second, a segment-level gradient loss preserves event structures. The 20 event bins are divided into pre- and post-event segments and summed within each segment to reduce bin-wise sparsity. Sobel gradients are computed on the segment maps. The gradient weight is defined as \[\mathbf{W}_{g}=1+\eta\|\nabla\mathbf{E}_{g}\|,\] and the structural consistency term is \[\begin{align} \mathcal{L}_{\mathrm{grad}} = \frac{ \sum \mathbf{W}_{g}\odot \rho(\nabla_x\mathbf{E}_{p}-\nabla_x\mathbf{E}_{g}) }{ \sum \mathbf{W}_{g}+\epsilon }\\ \quad+ \frac{ \sum \mathbf{W}_{g}\odot \rho(\nabla_y\mathbf{E}_{p}-\nabla_y\mathbf{E}_{g}) }{ \sum \mathbf{W}_{g}+\epsilon }. \end{align}\]

Third, a gradient-direction loss encourages coherent event structure: \[\mathcal{L}_{\mathrm{dir}} = \mathrm{mean}\!\left( \mathbf{W}_{g}\odot \left( 1- \frac{\nabla\mathbf{E}_{p}\cdot\nabla\mathbf{E}_{g}}{\|\nabla\mathbf{E}_{p}\|\,\|\nabla\mathbf{E}_{g}\|+\epsilon} \right) \right).\] In our implementation, \(\lambda_{\mathrm{rec}}=1.0\), \(\lambda_{\mathrm{grad}}=0.5\), and \(\lambda_{\mathrm{dir}}=0.1\). The event-domain supervision constrains the SNN event branch to produce clean and structurally reliable event representations, which improves the stability of component-level RGB–Event fusion.

3 Experiments↩︎

3.1 Experimental Details↩︎

3.1.1 Datasets↩︎

We evaluate the proposed RainDancer method on three video deraining benchmarks, including NTURain [9], RainSynComplex25 [13], and RainSynLight25 [16]. NTURain provides paired synthetic rainy/clean videos for full-reference evaluation and real-world rainy videos for generalization analysis. RainSynComplex25 contains 33 test sequences with dense and complex rain patterns, while RainSynLight25 contains 27 test sequences with relatively mild rain degradation. Since these benchmarks are frame-based, we generate the corresponding event streams using the v2e simulator [42], which enables paired RGB-Event evaluation under controlled settings. We further use RainVID&SS [25] to examine whether deraining benefits downstream object detection and semantic segmentation.

3.1.2 Training Details↩︎

Our RainDancer is implemented in PyTorch and trained on an NVIDIA RTX 3080 GPU. Each training sample contains three consecutive rainy RGB frames and the corresponding event voxel representation generated by v2e. The center frame is used as the restoration target. During training, paired RGB/event samples are randomly cropped into \(128\times128\) patches, and the temporal length is fixed to 3. We optimize the network using Adam with an initial learning rate of \(1\times10^{-4}\), \(\beta_1=0.5\), and \(\beta_2=0.999\). The batch size is set to 4, and the learning rate is decayed by cosine annealing.

3.1.3 Evaluation Metrics↩︎

For full-reference image quality evaluation, we report PSNR [43] and SSIM [44] on restored target frames. To assess perceptual quality in real-world scenes where clean references are unavailable, we use four no-reference image quality metrics: NIQE [45], BRISQUE [46], ILNIQE [47], and CLIPIQA [48]. In addition to low-level restoration metrics, we evaluate the influence of deraining on downstream perception. For object detection, we report mAP-based metrics [49], and for semantic segmentation, we report mIoU [50]. These metrics provide a more comprehensive evaluation of whether the restored videos are beneficial for subsequent visual understanding.

The training objective consists of RGB-domain restoration supervision and event-domain supervision. The RGB loss includes Charbonnier, edge, and SSIM-based terms. The event loss regularizes the predicted clean event representation using weighted reconstruction, structural gradient consistency, and gradient-direction consistency.

3.1.4 Baselines↩︎

We compare our RainDancer method with representative deraining approaches from different categories. FastDeRain [35] is included as a classical model-based video deraining method, and DRSformer [39] is used as a strong single-image deraining baseline. For RGB-based video deraining, we compare with DeLiVR [41], ESTINet [38], RainMamba [40], MFGAN [37], VDMamba [15], TMICS [36], J4RNet [13], and SpacCNN [9]. For event-guided deraining, we include MPEVNet [25], EGVD [24], UVD-Event [23], and EHN [26]. These methods cover model-based, single-image, RGB video-based, and RGB-Event deraining settings.

3.2 Experiments and Analysis↩︎

3.2.1 Comparison on Synthetic Datasets↩︎

We first evaluate the proposed method on NTURain, RainSynLight25, and RainSynComplex25. As reported in Table 1, our RainDancer obtains the highest PSNR on all three datasets. Compared with the second-best results, the PSNR gains are 0.66 dB on NTURain, 0.20 dB on RainSynLight25, and 1.42 dB on RainSynComplex25. The improvement is more pronounced on RainSynComplex25, where rain patterns are denser and more spatially complex. This result is consistent with the motivation of our RainDancer method: when RGB rain streaks, background textures, and motion-induced variations become difficult to distinguish, component-level RGB-Event interaction provides more explicit cues for rain/background separation.

We also report a lightweight variant, RainDancer-S. RainDancer-S shares parameters across progressive stages and uses a reduced channel width. It achieves performance comparable to strong RGB-Event baselines and obtains better results than MPEVNet on NTURain, while substantially reducing computational cost. This suggests that the proposed decomposition and component interaction strategy is not limited to a high-capacity model.

Fig. 4 shows qualitative comparisons on NTURain and RainSynComplex25. Several competing methods either leave residual streaks or introduce visible artifacts in regions with dense rain and background motion. In the NTURain example, some results contain color artifacts around rain-contaminated regions. In the RainSynComplex25 example, motion blur and incomplete rain removal are more evident. By contrast, our RainDancer method removes most rain streaks while maintaining sharper scene structures. These observations indicate that event cues are more effective when they are introduced after modality-specific rain/background decomposition, rather than being directly mixed with RGB appearance features.

3.2.2 Comparison on Real-world Datasets↩︎

We further evaluate real-world generalization on the real-world subset of NTURain. Since clean references are unavailable, Table 2 reports NIQE, BRISQUE, ILNIQE, and CLIPIQA. Our RainDancer method achieves the best BRISQUE, ILNIQE, and CLIPIQA scores, and ranks second on NIQE. DeLiVR obtains the best NIQE but is less competitive on BRISQUE and CLIPIQA. TMICS achieves a strong BRISQUE score, but its results tend to be smoother in detailed regions, as shown in Fig. 6. RainDancer-S performs slightly below the full model but remains competitive among both RGB-based and RGB-Event methods, indicating that the proposed design retains reasonable real-world robustness under a smaller model capacity.

Fig. 5 summarizes the no-reference evaluation by ranking all methods on each metric and averaging the ranks. This rank-based view reduces the bias caused by relying on a single no-reference metric. Our RainDancer method obtains the best average rank and remains among the top methods across all four metrics, suggesting balanced perceptual behavior on real-world rainy videos. The lightweight RainDancer-S has a weaker BRISQUE rank than the full model but keeps a competitive average rank, which further supports the scalability of the proposed design.

7pt

Fig. 6 presents visual results on real-world rainy videos. TMICS removes many rain streaks but also smooths local textures. DeLiVR reduces obvious rain degradation, while some short bright streaks remain. EGVD and MPEVNet benefit from event guidance, but residual artifacts are still visible in regions with background motion. Our RainDancer produces cleaner results with fewer remaining streaks and less texture smoothing. This behavior is consistent with the component-level interaction strategy: rain-related event responses are mainly used to support rain removal, whereas background-related representations are fused separately to preserve scene structures.

3.2.3 Model Complexity Comparison↩︎

Table 1 also reports model complexity in terms of parameters and FLOPs. FLOPs are measured on \(256\times256\) input patches, following the convention that one multiply-accumulate operation is counted as two floating-point operations. Our RainDancer contains 10.17M parameters, which is smaller than several strong RGB baselines such as ESTINet, MFGAN, and RainMamba. Its computational cost is higher than some lightweight RGB-only models because each progressive stage performs RGB-Event decomposition and component interaction. Nevertheless, its FLOPs remain lower than TMICS, ESTINet, and MPEVNet.

RainDancer-S provides a more efficient configuration. It uses 1.40M parameters and 318.74G FLOPs by sharing parameters across progressive stages and reducing the channel width. Compared with MPEVNet, RainDancer-S requires fewer parameters and reduces FLOPs by about 11.1 times, while maintaining comparable restoration accuracy and achieving better performance on NTURain. These results indicate that the proposed framework can be adjusted to different efficiency requirements without removing its core decomposition-interaction mechanism.

3.3 Ablation Study↩︎

All ablation variants are trained on NTURain using the same data split, optimizer, training schedule, and loss settings unless otherwise specified. We report PSNR and SSIM for quantitative comparison and provide visualizations to analyze how each component affects rain-background separation.

3.3.1 Fusion Strategy↩︎

We first study the RGB-Event interaction paradigm. Table 3 compares three variants: an RGB-only baseline, a direct RGB-Event fusion baseline, and the proposed decompose-before-interact design. Direct RGB-Event fusion improves PSNR from 30.28 dB to 35.64 dB, confirming that events provide useful motion-sensitive information for video deraining. However, its performance remains clearly below our full design. This gap indicates that event information is not sufficient by itself; the way it is introduced into RGB restoration is critical.

As shown in Fig. 7, the RGB-only baseline leaves visible rain residues, especially where rain streaks overlap with background textures. Direct fusion reduces part of the rain degradation but still suffers from residual streaks and local artifacts, which suggests that background-triggered events may interfere with RGB restoration when mixed without explicit separation. Our RainDancer produces cleaner restoration because rain/background components are first separated within each modality and then interacted between semantically corresponding components. This result directly supports the proposed decompose-before-interact paradigm.

3.3.2 Event Branch Processing↩︎

We next evaluate the design of the event branch. Table 4 compares a CNN event branch, a generic SNNformer event branch, and the proposed rain-oriented SNN branch. The CNN branch obtains 36.58 dB, indicating that a standard convolutional encoder can extract useful event features but has limited ability to model sparse temporal dynamics. Replacing it with SNNformer improves PSNR to 38.95 dB, showing the benefit of spike-driven temporal modeling. The proposed rain-oriented SNN further improves PSNR to 40.40 dB and SSIM to 0.9789, suggesting that adapting the event branch to rain-induced sparse and bursty responses is more effective than using a generic event encoder.

Fig. 8 visualizes the event features produced by different branches. The CNN branch gives diffuse responses and is easily affected by background-triggered events. SNNformer captures stronger activations but still responds to some non-rain structures. In comparison, the proposed rain-oriented SNN produces responses that are more concentrated on rain-related event patterns. This provides more discriminative event guidance for subsequent component-level RGB-Event interaction.

7pt

3.3.3 Loss Function↩︎

We then analyze the event-domain supervision. Event tensors are sparse and contain many inactive pixels; therefore, a conventional pixel-wise loss may be dominated by zero-valued regions and provide insufficient constraints on meaningful event structures. Table 5 compares several supervision choices. The standard \(\mathcal{L}_{1}+\mathcal{L}_{2}\) loss gives 39.12 dB. The sparsity-aware reconstruction loss \(\mathcal{L}_{\mathrm{rec}}\) improves PSNR to 39.56 dB, indicating that event sparsity should be explicitly considered. The gradient consistency loss and direction consistency loss also improve the result, showing that local event structure and orientation are useful constraints for rain-related event modeling.

Combining the three terms further improves PSNR to 40.06 dB. The weighted formulation achieves the best result, 40.40 dB and 0.9789 SSIM, because it balances event reconstruction, structural consistency, and directional consistency. This confirms that event supervision contributes not only as an auxiliary regularizer but also as a way to improve the reliability of event representations used in RGB restoration.

7pt

7pt

Fig. 9 reports the consistency between predicted background events and ground-truth events. A pixel is regarded as consistent if both the prediction and ground truth are zero or both are non-zero under a near-zero threshold. Compared with the alternatives, the proposed loss yields higher agreement with the ground-truth event distribution. This indicates that the weighted event-domain supervision better preserves sparse event structures and reduces unreliable event responses.

3.3.4 Fusion Design↩︎

We further study how RGB and event components should be fused. Table 6 reports four variants. Using only background fusion achieves 38.12 dB, suggesting that background compensation alone is insufficient for removing dynamic rain streaks. Rain-aware background fusion improves the result to 38.94 dB by reducing rain interference during background interaction. Adding rain fusion further increases PSNR to 39.61 dB, which shows that explicitly modeling the rain component is necessary for deraining.

The proposed event-guided rain fusion obtains the best result, 40.40 dB and 0.9789 SSIM. Compared with plain rain fusion, it uses event-derived rain responses to strengthen the localization and suppression of dynamic rain streaks. These results suggest that background and rain components play complementary roles: background fusion mainly supports structure preservation, whereas event-guided rain fusion contributes to rain removal.

3.3.5 Number of Progressive Stages↩︎

Finally, we evaluate the number of progressive stages. Table 7 shows that a one-stage model obtains 38.86 dB. Increasing the number of stages to two improves PSNR to 39.92 dB, and the three-stage model further reaches 40.40 dB. The gain from one to two stages is 1.06 dB, while the gain from two to three stages is 0.48 dB. This indicates that progressive refinement is beneficial, but the marginal improvement decreases as the number of stages increases.

The parameters and FLOPs increase approximately linearly because each stage contains a similar decomposition and interaction structure. Considering the trade-off between accuracy and computation, we adopt three stages in the full model and do not further increase the stage number.

Fig. 10 shows stage-wise rain prediction results. The one-stage model captures only part of the rain pattern and includes some irrelevant background responses. The two-stage model predicts more complete streaks, while the three-stage model gives a cleaner rain map with fewer background activations. This supports the use of progressive refinement for gradually improving rain-background separation.

7pt

3.4 Downstream Tasks↩︎

We further evaluate whether deraining improves downstream perception on RainVID&SS [25], which contains ImageNet-VID+ for object detection and CamVid+ for semantic segmentation. For fair comparison, all deraining models are retrained on RainVID&SS using the same experimental setting. We adopt YOLO-v3 [51] and MEGA [52] as object detectors, and ENet [53] and CGNet [54] as semantic segmentation models.

Table 8 reports both restoration and task performance. On ImageNet-VID+, our RainDancer method achieves the best PSNR/SSIM and the second-best detection accuracy for both YOLO-v3 and MEGA. MFGAN obtains slightly higher detection mAP, although its restoration metrics are lower than ours. This difference indicates that detection performance is not determined solely by full-reference restoration quality. Nevertheless, our RainDancer method improves the rainy input by a large margin and produces detection results close to or slightly higher than those obtained on clean frames under the tested detectors.

On CamVid+, our RainDancer method achieves the highest PSNR/SSIM and the best mIoU for both ENet and CGNet. The improvement is more evident for CGNet, where our RainDancer method reaches 62.14% mIoU. These results suggest that component-aligned RGB-Event deraining helps preserve semantic boundaries and local structures that are useful for segmentation.

0pt

Fig. 11 further analyzes the relation between restoration quality and downstream accuracy. Our RainDancer method achieves the highest PSNR and consistently strong perception results, but the plots also show that higher PSNR does not always lead to higher mAP or mIoU. For example, some methods with competitive PSNR still obtain limited segmentation accuracy. This observation suggests that downstream perception depends not only on pixel-level fidelity but also on the preservation of semantic boundaries and task-relevant local structures. The proposed component-aligned interaction is beneficial in this respect because it reduces rain interference while maintaining background structures.

Fig. 12 shows qualitative object detection results. In the YOLO-v3 example, the restored frame from our RainDancer method enables the detector to identify a marginal bicycle that is missed in several competing results. In the MEGA example, most methods detect the main objects, but our result yields higher confidence scores. These results indicate that deraining can improve object-level perception when the restored image preserves discriminative contours and local structures.

Fig. 13 presents semantic segmentation results on CamVid+. Compared with the competing methods, our restored images lead to more coherent semantic regions and clearer object boundaries for both ENet and CGNet. Several baselines either leave rain-induced disturbances or over-smooth local structures, which causes fragmented or unstable segmentation predictions. The results support the quantitative findings in Table 8: our RainDancer method improves restoration quality while better retaining structures that are useful for high-level visual understanding.

4 Conclusion↩︎

This paper presents a progressive RGB–Event video deraining framework following a decompose-before-interact principle. Instead of directly fusing heterogeneous RGB and event features, the proposed method first decomposes each modality into rain and background components and then conducts component-level interaction between semantically aligned representations. A rain-oriented SNN event branch is developed to capture sparse and bursty rain dynamics, while rain-aware background fusion and event-guided rain fusion are introduced to reduce cross-modal interference and improve rain removal. Moreover, an event-domain supervision strategy is designed to constrain sparse reconstruction, structural consistency, and gradient orientation of event representations. Experiments on synthetic and real RGB–Event video deraining datasets show that the proposed method outperforms existing RGB-based and event-guided approaches in quantitative metrics, visual quality, and downstream perception robustness. The results demonstrate that explicit rain-background decomposition and component-aligned RGB–Event interaction provide an effective solution for reducing rain-background entanglement in dynamic rainy videos.

References↩︎