1 Introduction↩︎

RGB-Thermal infrared (RGB-T) object detection aims to leverage the complementarity from different modalities to handle various challenging conditions [1], [2], drawing much attention in recent years. Numerous benchmarks [3]–[5] and detectors [2], [6], [7] have been proposed. Detectors based on feature fusion have acquired superior detection performance [8], which typically extract the RGB and thermal infrared (TIR) features using two independent backbones and perform pixel-to-pixel feature fusion [9], [10]. However, the computational complexity of these powerful RGB-T detectors remains undesirably high, reaching up to hundreds of GFLOPs [8], [11]–[13], restricting their practical applicability.

Existing powerful RGB-T detectors primarily employ heavy dual backbones and perform sophisticated feature fusion uniformly across the entire image to detect the target objects accurately. Nevertheless, the backgrounds, such as the sky and ground, which generally occupy the majority of the image, are smooth and can be readily distinguished by lightweight models, even when utilizing only a single modality. Hence, processing the backgrounds with a heavy feature extractor and a complex fusion mechanism is unnecessary and highly inefficient. In light of this, an effective strategy for constructing lightweight yet high-precision RGB-T detectors is to pre-filter these simple background regions using compact single-modality models and employ a powerful cross-modality fusion module to process the remaining regions. This is essentially an efficient sparse fusion mechanism, adaptively allocating more computational resources to potential foregrounds, as shown in Figure 1 (a).

The prerequisite for the sparse fusion mechanism is that the lightweight model can identify potential foregrounds at a high recall rate without unacceptable mistaken drops. To figure this out, we conduct an analytical experiment with YOLOv8-Large and YOLOv8-Small [14]. Figure 1 (b) presents the Precision–Recall (PR) curves on FLIR [3]. We observe that the lightweight detector achieves a cut-off recall comparable to that of the heavyweight detector, both remarkably close to 100%, when the IoU threshold is 50%. Such a high recall validates the effectiveness of a lightweight model for searching potential foregrounds, i.e., proposals. This demonstrates the feasibility of the sparse fusion mechanism. Nevertheless, we observe two limitations from the PR curves. First, the PR curve of the lightweight detector remains consistently lower than that of the heavyweight one at IoU=50%. This suggests a high false-positive rate in the preliminary proposals, which therefore requires further verification for accurate recognition. Second, the recall drops sharply at IoU=75%, indicating the limited localization accuracy of the lightweight model. These two issues can be effectively mitigated with the robust features derived from cross-modality fusion.

In this paper, we instantiate Sparse Fusion mechanism and propose an Efficient RGB-T object Detector, dubbed SFEDet, as illustrated in Figure 1 (d). Specifically, we employ two lightweight detectors, one for each modality, serving as region proposal networks (RPNs) to identify potential foreground regions, i.e., Regions of Interest (RoIs), for further detailed examination. Herein, to avoid missing real objects during proposal extraction, we construct the proposal set by taking the union of RoIs from individual modalities. Subsequently, we borrow RoI Align [15] to extract paired RoI features from the two modalities. Despite the high recall of the collected proposals extracted by our RPNs, as shown in Figure 1 (c), the RPNs still suffer from false positives and inaccurate localization. To accurately filter out false positives and refine coarse bounding boxes, we introduce a Fusion-driven Examination and Refinement (FER) module to process the RoIs for final detection, which performs multi-step feature fusion followed by step-wise classification and box refinement. In this way, the lightweight RPNs process the entire image preliminarily, while the sophisticated FER module operates only on sparse RoI features with fixed spatial sizes. This design enables a favorable balance between efficiency and performance. Moreover, due to the sparse fusion mechanism, SFEDet demonstrates excellent scalability for high-resolution images, as the computational complexity of FER depends solely on the number of RoIs, as demonstrated in Figure 1 (e).

The implementation of the FER module is crucial to the efficiency and performance of our SFEDet. To this end, we introduce a Rolling Convolution operator into the FER module to replace the conventional convolution operator, improving its efficiency. Furthermore, we adopt a denoising training strategy to facilitate the FER module to learn to process proposals with large localization errors, consequently improving its detection performance. We conduct elaborate experiments on FLIR, M3FD, and LLVIP datasets, where the SFEDet achieves a balance between efficiency and precision. The main contributions can be summarized as follows:

• We introduce a sparse fusion mechanism that adaptively allocates greater computational resources to potential foreground regions; it is then instantiated within our SFEDet framework, enabling SFEDet to achieve a favorable balance between efficiency and performance.

• We propose a fusion-driven examination and refinement method to process the sparse proposals, which effectively filters out the false positives and refines the coarse bounding boxes via step-wise feature enhancement.

• We conduct extensive experiments on popular RGB-T detection benchmarks, demonstrating the effectiveness of our method and highlighting its scalability for high-resolution images.

2 Related works↩︎

2.1 RGB-T Object Detection↩︎

In recent years, several benchmarks [3], [16] have been established and promote the development of the detection area. Some of them, such as M3FD [4] and LLVIP [5], are well aligned and adopted not only in object detection but also in pixel-level image fusion.

The algorithms for RGB-T object detection have advanced substantially, evolving from image fusion-based methods [17], [18] and box fusion-based methods [19] to feature fusion-based approaches. Among these, He et al. [20] proposed weighting different modalities according to the information difference measured within their local neighborhoods. Li et al. [6], [21] adjusted the fusion weights of RGB and TIR features based on illumination conditions. Hu et al. [22] incorporated edge priors to constrain the fusion process, while ICAFusion[8] employed a cross-attention mechanism for modality guidance. Other studies [2], [23] used box-aligned masks as spatial guidance to achieve more precise fusion. Chen et al. [24], [25] introduced vision transformers into RGB-T object detectors, while Dong [26] employed Mamba [27] to perform feature fusion. In addition, several studies have addressed specific challenges in RGB-T object detection. Tian et al. [10], [28] focused on the non-alignment issue, whereas Zhou et al. [7], [29] investigated the modality imbalance problem, where the RGB and TIR modalities contribute unequally to the fusion results. However, the computational overhead of feature fusion-based RGB-T detectors has received little attention to date.

2.2 Efficient Object Detectors↩︎

Efficient lightweight detectors have been extensively studied in general object detection. The YOLO [30] series is well known for enabling real-time detection. Early versions [31], [32] are anchor-based, which makes both training and deployment relatively complex. Subsequent improvements, such as YOLOX [33] and TOOD [34], introduce dynamic sample assignment, a technique that has been widely adopted in later versions of YOLO [14], [35]. YOLOv7 [36] and PP-YOLOE [37] further enhance the network architectures to improve inference speed and overall efficiency. YOLOv10 [35] focuses on eliminating the non-maximum suppression (NMS) post-processing step, which has long been considered a bottleneck in detector acceleration. DETR-like detectors [38]–[40] generally achieve higher upper-bound performance but at the cost of increased computational complexity. RT-DETR [41] is a real-time DETR-based detector that strikes a favorable balance between speed and accuracy.

Although efficient single-modality detectors have been extensively explored, the modal-specific and unshared feature extraction pipelines, together with the subsequent fusion process, remain computationally demanding. This issue has received limited attention in prior studies. Deevi [42] attempts to alleviate the training cost by freezing two backbones; however, the overall computational complexity of the model remains unchanged. Addressing this limitation constitutes the primary goal of our work.

3 Method↩︎

In this section, we present the instantiation of the sparse fusion mechanism, i.e., the SFEDet framework, which performs efficient and accurate RGB-T detection through two stages: 1) the first stage takes a quick look at the entire image with lightweight model to filter out the redundant backgrounds while retaining potential foreground regions; 2) the second stage performs detailed examination and refinement within the sparse foreground regions, i.e., the RoIs, to produce the final detection results. Next, we describe the overall framework of SFEDet and detail its key components.

3.1 Overall framework of SFEDet↩︎

Figure 2 illustrates the overall framework of SFEDet performing detection in two stages. In the first stage, we employ two lightweight detectors serving as independent RPNs to filter out the apparent background regions through preliminary detection. We then construct the proposal set by taking the union set of proposals from individual modalities. To further reduce redundancy in the preliminary detections, we subsequently apply NMS in the union set to get sparse, non-overlapping proposals. With these proposals, we extract the corresponding RGB-T RoI feature pairs from the multi-level FPN features via RoI Align, where each proposal box is used to get spatially aligned features from two modalities. The proposals generated in the first stage typically contain numerous false positives and poorly localized boxes. To address these issues, we further introduce a Fusion-driven Examination and Refinement (FER) module in the second stage. It fuses the RGB and TIR features through multi-step feature enhancement, producing a more robust representation for each RoI and facilitating the suppression of false positives. For proposals with inaccurate localization, a step-wise refinement strategy is adopted to progressively refine the bounding boxes.

The most computationally intensive component of our SFEDet, i.e., the FER module, is activated only within potential foreground regions. This means that the computational cost depends primarily on the number of RoIs. Consequently, SFEDet exhibits strong scalability when processing high-resolution images. We provide detailed designs of the proposed FER module, the box refinement procedure, and the overall training schedule of SFEDet in the following sections.

3.2 Fusion-driven Examination and Refinement↩︎

Due to numerous false positives or poorly localized proposals in the first-stage results, we thus design an elaborate Examination and Refinement module based on the cross-modality feature fusion to address this, as illustrated in Figure 2. We deploy an efficient differential-enhanced group-wise fusion (DEGF) to balance the fusion cost and quality. The following multi-step enhancements with auxiliary features ensure a robust representation for each RoI.

Differential-enhanced group-wise fusion. This module contains a group-wise cross-modality fusion and a subsequent differential enhancement module. Before fusion, the RoI features from both RGB and TIR modalities are simultaneously extracted from three stages of the feature pyramid network (FPN) via RoI Align, denoted as \(\mathbf{F}_{fpn-i}^{M}\), where \(M \in \{rgb, tir\}\) and \(i \in \{1,2,3\}\). Multi-stage features from FPN are then reduced into a cross-stage representation \(\mathbf{F}_{fpn}^{M} \in \mathbb{R}^{C \times 5 \times 5}\) for each proposal, where \(C\) denotes the number of channels and 5 represents the RoI spatial size.

Core of this module is the group-weighting fusion applied to \(\mathbf{F}_{fpn}^{rgb}\) and \(\mathbf{F}_{fpn}^{tir}\), as illustrated in Figure 2. After the \(N_{gw}\)-group split along the channel dimension, the weights within each group are predicted separately and shared across the entire group in fusion, which reduces the computational cost. The softmax normalization ensures that the sum of weights equals one for each channel pair. The fused features are denoted as \(\mathbf{F}_{fu}^{1}\).

The differential enhancement module aims to adaptively learn a supplementary representation for each modality by comparing the fused features \(\mathbf{F}_{fu}^{1}\) with the original modality-specific features. The adaptive weights are learned from the joint representation of \(\mathbf{F}_{fu}^{1}\) and \(\mathbf{F}_{fpn}^{M}\), where the feature difference is adopted as a computationally efficient operation. Compared with the Add and Concatenation operation, the adopted difference operation is lighter in computation than the Concatenation while achieving superior performance to Add, as demonstrated in the ablation study. As illustrated in Figure 2, the enhancement procedure is formulated as: \[\label{eq:diff} \begin{alignedat}{1} \mathbf{\hat{F}}_{rgb} &= \mathbf{F}_{fpn}^{rgb} \cdot \Psi_{rgb}(\mathbf{F}_{fu}^{1} - \mathbf{F}_{fpn}^{rgb}), \\ \mathbf{\hat{F}}_{tir} &= \mathbf{F}_{fpn}^{tir} \cdot \Psi_{tir}(\mathbf{F}_{fu}^{1} - \mathbf{F}_{fpn}^{tir}), \\ \mathbf{F}_{fu}^{2} &= \phi_{LN}(\mathbf{F}_{fu}^{1} + \mathbf{\hat{F}}_{rgb} + \mathbf{\hat{F}}_{tir}), \end{alignedat}\tag{1}\] where \(\Psi(\cdot)\) is the convolution block, and \(\phi_{LN}\) means the Layer Norm.

Auxiliary Feature Enhancement. Since the FPN features are high-level representations extracted from later stages of the network, we introduce low-level features from the backbone (BB) as auxiliary (Aux.) inputs to enhance the fusion representation. The fusion procedure is illustrated in Figure 2, where auxiliary features are re-captured by RoI Align from the backbone feature maps.

Rolling Convolution. To further reduce computational costs, we propose a variant of Group Convolution (GC), termed Rolling Convolution, which replaces all conventional convolutional operators in the FER module. The performance of cascaded GCs tends to be suboptimal due to the lack of inter-group interactions. To address this issue, we deploy an additional counterpart of GC, where the features are rolled along the channel dimension before convolution. The rolling shift is set to half of the channels within each group, as illustrated in Figure 2. A subsequent point-wise convolution merges the outputs from the two GCs. Compared with conventional convolution, the proposed Rolling Convolution employs fewer parameters while providing greater depth and nonlinearity for the detector.

3.3 Step-wise Box Refinement↩︎

Step-wise Refinement. Owing to the poorly localized bounding boxes in the outputs of lightweight RPNs, we adopt a step-wise refinement strategy to progressively refine the bounding boxes. In the FER module, we employ a two-step refinement scheme, where the bounding boxes are predicted and decoded conditioned on the predictions from the previous step. The residuals between successive bounding boxes are predicted based on the updated fusion features, meaning that the auxiliary features are re-captured via RoI Align at the end of the first step, once the bounding boxes are updated.

Box Encoder and Decoder. The RPNs in our framework adopt the distance-based decoder to predict bounding boxes for better performance. However, in the FER module, the RoI features lose the stride-scale information, which is inherently preserved in conventional multi-scale feature maps and indicates the actual scale of normalized boxes in the original image space. To address this issue and unify the box constraint format across all heads, we leverage the scale of the proposals to normalize and decode the bounding boxes in the FER module. Let \(d_i\) denote the predicted distance of a side, and \(L\) denote the theoretical maximum distance that the prediction head can output; the bounding boxes are then decoded as: \[\label{eq:decode} D_i= \left\{ \begin{array}{ll} \frac{x_{2} - x_{1}}{L} \cdot d_i, & i \in \{left, right\}, \\ \frac{y_{2} - y_{1}}{L} \cdot d_i, & i \in \{upper, bottom\}, \end{array}\right.\tag{2}\] where \(D_i\) is the updated distance of each side to the center of the proposal box after decoding, and \([x_1, y_1, x_2, y_2]\) indicates the position of the proposal box. The corresponding encoding procedure is then formulated as follows: \[\label{eq:encode} \hat{d}_i= \left\{ \begin{array}{ll} \frac{L}{x_{2} - x_{1}} \cdot D_{i}^{gt}, & i \in \{left, right\}, \\ \frac{L}{y_{2} - y_{1}} \cdot D_{i}^{gt}, & i \in \{upper, bottom\}, \end{array}\right.\tag{3}\] where \(D_{i}^{gt}\) is the distance of each side of the ground truth bounding box to the proposal center, \(\hat{d}_i\) is the target of the prediction set in the loss function.

3.4 Denoising Training strategy↩︎

The proposed FER module is trained using the proposals generated by the RPNs. However, during the early stages of training, the RPNs may not be sufficiently robust to cover all regions containing objects. As a result, some regions that potentially contain objects are dropped, which often correspond to hard samples that are crucial for learning. To mitigate this issue, we adopt a denoising training strategy, where \(N_{de}\) groups of noisy variants of the ground-truth bounding boxes are added to the proposal list. This strategy enhances the diversity of training samples and encourages the FER module to learn from challenging cases. The procedure of noise injection is illustrated in Figure 2 and formulated as: \[\label{eq:denoise} \begin{alignedat}{1} (x_i, y_i) &\rightarrow (x_i +\delta_x, \;y_i + \delta_y), i \in \{1, 2\}, \\ \delta_x &\sim U(-0.4*W, +0.4*W), \\ \delta_y &\sim U(-0.4*H, +0.4*H), \end{alignedat}\tag{4}\] where W and H denote the width and height of the GT boxes, respectively. The assigner automatically categorizes these noisy proposals into positive and negative samples and matches them with the corresponding GT boxes.

3.5 Training Procedure↩︎

Two-stage training. Our method employs a two-stage training strategy to ensure stable and robust convergence. In the first stage, two independent RPNs are trained separately for the RGB and TIR modalities, and their weights are then initialized in the detector as a warm-up. In the second stage, the entire detector is trained in an end-to-end manner, where the parameter updates of the two RPNs are regulated by a learning rate decay factor \(\lambda\).

Total Loss. The total loss of our architecture is composed of losses from two independent RPNs and two-step refinements in the FER module. The loss configuration for each head of the FER module can be unified with that of the RPNs based on our encoding scheme proposed in Sec 3.3. Consequently, all four heads follow the standard loss settings of YOLOv8, where binary cross-entropy (BCE) loss is used for classification, and CIoU and DFL [43] losses are adopted for bounding box prediction. The overall loss function is defined as follows: \[\label{eq:total95loss} \mathcal{L}_{total} = \mathcal{L}_{rpn}^{rgb} + \mathcal{L}_{rpn}^{tir} + \mathcal{L}_{fer}^{(1)} + \mathcal{L}_{fer}^{(2)},\tag{5}\] where each loss item is composed of \[\mathcal{L} = \alpha \mathcal{L}_{BCE} + \beta \mathcal{L}_{CIoU} + \gamma L_{DFL}.\] The \(\alpha\), \(\beta\), and \(\gamma\) in Eq 5 are the weights of different loss terms.

4 Experiments↩︎

4.1 Experimental Setup↩︎

Datasets. M3FD [4] is a well-aligned RGB-T dataset widely used in RGB-T object detection and image fusion. It contains 6 categories and is built with 4.2K image pairs at 1024\(\times\)​768 resolution. The dataset contains several challenging scenarios with extreme light and weather conditions, and objects of different scales, making it hard for any single modality. The relabeled FLIR [3] benchmark is a large-scale dataset widely adopted in RGB-T object detection, including 3 categories at 640\(\times\)​512 resolution. The LLVIP [5] is a high-quality RGB-T pedestrian detection dataset with more than 15K image pairs at 1280\(\times\)​1024 resolution.

Evaluation metrics. We use the standard COCO-style mAP 0.5:0.95 (denoted as mAP) and AP50 as the detection metric throughout all experiments.

Implementation details. We deploy the YOLOv8-Small as the RPN in all experiments. The \(N_{gw}\) in group-wise fusion is 4, and the \(N_{de}\) in denoising training is 10 to balance the training cost. The \(L\) in the box-decoding is 16, as that in standard YOLOv8. The groups in the rolling convolution and the RoI size are detailed in the ablation study. The weights of loss items are set to \(\alpha=0.5\), \(\beta=7.5\), and \(\gamma=0.375\). We adopt the AdamW optimizer for training, with a learning rate of 0.0002 and a batch size of 6. The learning rate decay for the RPNs, i.e., the \(\lambda\), is set to 0.005. Please refer to Appendix B for more details.

4.2 Comparisons with State-of-the-art Methods↩︎

Results on M3FD dataset. The quantitative comparison on the M3FD dataset is presented in Table 1. The term CSP in the table refers to the CSP-Darknet used in the YOLOv5/v8 series. We re-evaluated several methods using their official code to measure their computational cost at a resolution of 1024×768, which are marked with \(^\dagger\). Our method achieves competitive detection accuracy while requiring only about 30% of the computational cost compared to other approaches on the high-resolution images.

Results on FLIR and LLVIP datasets. Table 2 presents comparison results on them. The proposed SFEdet achieves precision comparable to significantly heavier detectors. On the LLVIP dataset, the image area increases by 400%, leading to a substantial rise in computational cost. SFEDet experiences a 2.4\(\times\) increase in FLOPs, which is smaller than that of other methods, for example, 3.8\(\times\) in EI2Det, demonstrating the advantage of our method in maintaining sub-linear growth in computational complexity. Figure 3 presents a qualitative comparison between COFNet and our SFEDet in challenging illumination conditions, demonstrating the effectiveness of our sparse fusion architecture. More qualitative detection results on different datasets are in Appendix B.

Furthermore, we construct a lighter SFEDet version for the FLIR and LLVIP datasets by filtering proposals and increasing sparsity, denoted as SFEDet-Lite. We set the configuration to limit the AP50 decrease within 0.1, presented in Table 2, which achieves an additional reduction of approximately 10G FLOPs.

4.3 Ablation Study↩︎

Comparison with other fusion manners. To quantitatively analyze the performance and complexity of our sparse fusion detection framework, we construct several RGB-T detector prototypes using different fusion strategies, as summarized in Table 3. We perform dense fusion on image pairs, backbone features, and FPN features via channel-wise concatenation. Given the relatively limited performance of the image-level fusion detector, we deploy the Large model for comparison. Furthermore, we apply the proposed DEGF module in a dense fusion manner to further demonstrate the efficiency of the sparse fusion mechanism. Experimental results on the M3FD and FLIR datasets indicate that our method achieves a favorable trade-off between computational cost and performance.

Effects of model components. The ablation study of different modules is summarized in Table ¿tbl:tab:ablation?. The results indicate that the group-wise fusion provides better performance due to fine-grained adaptation in fusion, without any increased cost. Compared with using element-wise addition, the differential enhancement module improves precision without increasing computational cost. The auxiliary feature column demonstrates the necessity of the third-stage feature enhancement, where low-level features extracted from the backbone are more effective than high-level features from the FPN. The second-refinement column illustrates the improvement introduced by the step-wise refinement mechanism. All convolutional layers in the FER module are replaced by rolling convolution layers, which reduce computational complexity while enhancing performance. This may be attributed to the increased depth of the network. Without the denoising training strategy, the detector performance declines, possibly because the model lacks sufficient hard training samples.

Effects of different convolution operations. Lightweight variants of the conventional convolutional layer have been widely studied, including group convolution, depthwise (DW) convolution, and the ShuffleNet [44] block. We perform a comparative study among several existing lightweight convolution designs, as summarized in Table 4. The results indicate that all lightweight convolutions achieve performance comparable to other detectors, while the proposed rolling convolution further improves performance at slightly higher computational cost.

Total Effects of FER. The experimental results demonstrate the effectiveness of FER in improving localization quality, as shown in Table ¿tbl:tab:ablation?. The mAP of the baseline increases largely in the FLIR and M3FD. The qualitative experiment on the typical conditions shown in Figure 4 demonstrates the effectiveness of FER. The quantitative results show that the false-positive count is reduced from 26.5 to 2.1 per image on M3FD. More details are in Appendix A.

To evaluate the robustness of FER on poor localized proposals, we progressively noise the proposals to ensure that their IoUs are below the predefined upper bound (from 1.0 to 0.5) to simulate providing poor proposals for FER. The corresponding results are reported in Table ¿tbl:tab:ppl95iou?, where all results are obtained without retraining, and demonstrate the robustness of our SFEDet. This robustness owes to 2 designs of SFEDet: 1) the proposed denoising training strategy; and 2) the scale factor of 1.1 to enlarge the proposals before pooling.

RoI size and Rolling Convolution groups. The feature size obtained from RoI Align and the group configuration in the rolling convolution are two key hyperparameters in our method, as they correspond to detection performance and computational complexity. We conduct an additional ablation study to investigate their effects, as summarized in Table 5. Configurations highlighted in blue represent the balanced setting that achieves a desirable trade-off between cost and performance and are thus adopted in the final model configuration.

4.4 Complexity Analysis↩︎

FLOPs vs. Sparsity. The low computational cost of our SFEDet primarily results from its sparse fusion and prediction mechanisms. Consequently, the prediction sparsity has a significant impact on the overall computational cost. We further investigate the relationship between FLOPs and prediction sparsity, which we define as: \[Sparsity = n \cdot (S_{roi})^2 \;/ \sum_{level=1}^{3} w_{fpn}^{level} \cdot h_{fpn}^{level},\] where \(n\) denotes the number of proposals, \(S_{roi}\) is the RoI feature size, and \(w_{fpn}\) and \(h_{fpn}\) represent the feature map widths and heights at different levels, respectively. This metric represents the ratio of the spatial area where convolution-based computations are performed. We adjust the sparsity by varying the threshold of RPN outputs, as illustrated in Figure 5. The curves indicate that the computational cost introduced by the FER is linearly correlated with the number of proposals. Experimental results on different datasets suggest that our detector maintains high precision even under large sparsity, while substantially reducing the overall computational cost. In high-resolution images, most of the computation is dominated by the RPN, whereas the FER accounts for only a small fraction of the total cost. The qualitative results in Figure 6 visualize the spatial distribution of computational complexity, showing that the computation is sparse with most background regions excluded.

FLOPs vs. Image size. We further examine the relationship between computational complexity and image area within our framework. We respectively evaluate the RPN module, SFEDet, and its dense variant with DEGF on datasets of varying resolutions. Quantitative results are presented in Figure [fig:floaps95area]. The RPN module and SFEDet-Dense are fully convolutional networks, and their computational costs scale linearly with the detection area. Our SFEDet achieves improved detection performance with only a modest increase in FLOPs, and exhibits sub-linear growth in computational cost with respect to detection area.

FPS vs. Number of GTs. The FPS of our SFEDet is related to the number of RoIs, i.e., the objects. The quantitative evaluation results are shown in Figure 7. Our method achieves a comparable FPS to those large dense detectors [22], [23], while largely outperforming them in samples with fewer objects.

4.5 Discussion on spatial misalignment↩︎

The spatial shifts between the RGB and TIR images lead to a misalignment in feature fusion, which impacts both the dense fusion detector (e.g., EI2Det [22]) and our SFEDet, as shown in Figure [fig:shifts]. In recent research [10], some specialized techniques are developed to handle this issue. We adopt the corresponding data augmentation, i.e., the shift, in the training of our SFEDet. The impact of misalignment can be effectively alleviated, as shown by the red line in Figure [fig:shifts], demonstrating the practical effectiveness of our sparse fusion architecture.

5 Conclusion↩︎

In this work, we propose an efficient RGB-T detector, SFEDet. By filtering most background at the very beginning, the SFEDet performs feature fusion-driven examination and step-wise refinements via the proposed FER module only within potential RoIs, thereby improving the computational efficiency. We introduce a Rolling Convolution operator and a denoising training strategy to reduce computational cost and improve the performance of the FER module, respectively. We believe that SFEDet may contribute to the development of efficient RGB-T object detection.

Acknowledgments↩︎

This work was supported in part by the Guangdong Basic and Applied Basic Research Foundation (Grants No. 2025A1515010705).

Appendix↩︎

6 Effect of the proposed FER↩︎

We conduct a comparison to evaluate the effectiveness of the proposed FER module, as shown in Table 6. We adjust the score threshold to maintain the same recall rate as the RPNs and count the false positives in each image pair. The results demonstrate that FER more effectively suppresses false-positive proposals generated by the RPNs.

7 Additional Details of Implementation↩︎

In this section, we detail the implementation and training phase. In the construction of the proposed SFEDet, we try to balance the computational cost and the detection performance. Each component in the SFEDet and the FER module is carefully designed. The parameters of each component are detailed in Table 7.

Two independent RPNs are two standard YOLOv8-Small [14] detectors, with multi-class classification retained to improve feature quality. The DEGF module, as mentioned in the Method, is a high-efficiency, lightweight fusion module. Two detection heads in the FER module are the heaviest modules in our framework. The heads share the same setting as those in YOLOv8, which adopt decoupled heads. Each head contains two independent prediction branches, i.e., the classification branch and regression branch. Though the number of channels in our SFEDet is set to 128 in most modules, the dimensions of the channels in the two heads are set to 512 for better performance. Though the prediction head is heavyweight, the total efficiency is guaranteed because of the sparse fusion.

We adopt several techniques in training. First, we deploy the Exp-Momentum Exponential Moving Average (ExpMomentumEMA) [33] to achieve a better convergence in thermal-related datasets, where the momentum is set to \(1e-4\). At the beginning of training, we deploy a linear warm-up scheduler for the learning rate for 500 training iterations. Then, we decay the learning rate by a 0.1 factor after different training epochs for different datasets for better convergence. Borrowing from literature [7], we randomly add noise to one of the modalities during RGB and TIR. The type of noise is randomly selected during Gaussian, Contrast, and Blur, following a uniform distribution. These data augmentations enhance the diversity of samples, thus improving the robustness of features after fusion and overcoming the modality imbalance issue [7].

Training loss analysis. The loss curves of SFEDet on different datasets are shown in Figure 8. Across all datasets, the losses of the FER heads exhibit a sharp decrease at the beginning, as the FER module is trained from scratch. During the later training stages, the deeper heads tend to converge to lower loss values under the same evaluation criteria, which suggests an improvement in the representational capacity of the detector. Furthermore, the differences between the losses of the two RPNs reflect the distinct contributions of the RGB and TIR modalities. In the M3FD [4] dataset, the RGB and TIR modalities tend to contribute comparably to the overall detection performance, owing to their complementary sensitivities to different object categories, such as traffic lights (in RGB) and persons (in TIR). In contrast, for the FLIR [3] and LLVIP [5] datasets, where the primary target objects (i.e., persons and cars) are generally more salient in TIR images, the TIR modality plays a more dominant role. The comparison between RPNs and the final heads demonstrates the effectiveness of the proposed cross-modality sparse fusion and examination.

8 More Qualitative Detection Results↩︎

We present more qualitative detection results of our SFEDet. The results from different datasets are respectively shown in Figure 9, Figure 10 and Figure 11. The first row of each figure is the RoIs obtained from RPNs, where the preliminary detections are processed by the NMS to get the sparse proposals. The second row shows the distribution of computational complexity. Considering that some regions are repetitively computed, these regions suffer a heavier complexity, highlighted with the hot color. The third row shows the final detection bounding boxes, where the boxes of different colors mean different categories.

The sparsity of RoIs demonstrates the effectiveness of our SFEDet. Due to the poor imaging quality in RGB images of the FLIR dataset, the proposals obtained from the RGB images are slightly more than those from other datasets. Meanwhile, in the LLVIP dataset, due to the lower challenging conditions, most of the background areas are pre-filtered, avoiding the fusion and heavyweight inference module. In the second stage, the RoIs are carefully examined and refined to find the sparse bounding boxes. The difference between RoIs and the final results shows the achievement of our proposed fusion-driven examination and refinement module, which excludes the false positives in proposals and modifies the inaccurate bounding boxes.

References↩︎