1 Introduction↩︎

1.1 Background↩︎

VISUAL environment perception (VEP) serves as a fundamental and front-end module in robotic systems [1]. Semantic segmentation and stereo matching are two key VEP tasks [2]. The former, akin to the ventral stream in our brain, provides a pixel-level understanding of the scene, while the latter, akin to the dorsal stream in our brain, mimics human binocular vision to acquire accurate depth information, essential for 3D geometry reconstruction [3]. These two tasks collaborate to deliver both contextual and geometric information, resulting in a comprehensive scene understanding that significantly enhances the capabilities of robotic systems [4].

Nevertheless, previous studies [5]–[9] address these two tasks with separate networks, which limits their potential to share informative contextual and geometric information [10]. For instance, stereo matching networks can occasionally produce ambiguous disparity estimations, particularly in texture-less and occluded regions [11]. Semantic segmentation can provide pixel-level scene understanding to disambiguate these cases, ultimately leading to more reliable disparity estimations [12]. In addition, semantic segmentation networks often struggle to distinguish clear object boundaries, particularly in complex driving scenarios, due to the lack of spatial geometric information [13].

1.2 Existing Challenges and Motivation↩︎

Taking our previous work S\(^3\)M-Net [14] as an example, its adopted feature fusion strategy essentially performs an element-wise summation between contextual and geometric feature maps at each stage. These feature maps are then directly fed into subsequent layers without excluding irrelevant information, leading to the loose coupling between the semantic segmentation and stereo matching tasks within the encoder.

1.3 Contributions↩︎

Therefore, in this article, we introduce Tightly-Coupled Semantic Segmentation and Stereo Matching Network (TiCoSS), an end-to-end joint learning approach that focuses primarily on improving the coupling between stereo matching and semantic segmentation, which has not been emphasized in previous related studies. Building upon our previous work S\(^3\)M-Net, our proposed TiCoSS introduces three new techniques: (1) a Tightly-Coupled, Gated Feature Fusion (TGF) strategy, which utilizes a series of Selective Inheritance Gates (SIGs) to propagate useful contextual and geometric information from the preceding layer to the current layer; (2) a Hierarchical Deep Supervision (HDS) strategy that uses the fused features with the highest resolution to guide deep supervision throughout each branch, as these features contain the most abundant local spatial details; (3) a novel Coupling Tightening (CT) loss, consisting of a widely used stereo matching loss presented in [9], the Semantic Consistency-Guided (SCG) loss in [14], a Disparity Inconsistency-Aware (DIA) loss that utilizes disparity estimations to refine segmentation results, and a Deep Supervision Consistency Constraint (DSCC) loss which employs the Kullback-Leibler (KL) divergence to improve prediction consistency across outputs from all deep supervision branches. These contributions collectively advance S\(^3\)M-Net, resulting in TiCoSS, a new, powerful, and tightly-coupled joint learning framework that enables robust and accurate simultaneous semantic segmentation and stereo matching. Extensive experiments conducted on the vKITTI2 [15] and KITTI 2015 [16] datasets unequivocally demonstrate the effectiveness of the aforementioned contributions and the superior performance of TiCoSS over other state-of-the-art (SoTA) approaches.

In summary, the main contributions of this article include:

• The TGF strategy, which propagates useful contextual and geometric information from the preceding layer to the current layer, enabling more effective feature fusion for semantic segmentation;

• The HDS strategy, which uses the fused features with the richest local spatial details to guide deep supervision across each branch;

• The DIA loss and the DSCC loss that tighten the coupling between the two tasks, thereby further improving semantic segmentation performance.

1.4 Article Structure↩︎

The remainder of this article is organized as follows: Sect. 2 reviews related prior arts. Sect. 3 introduces our proposed TiCoSS. Sect. 4 compares our network with other SoTA approaches and presents the ablation studies.

2 Literature Review↩︎

2.1 Semantic Segmentation↩︎

Semantic segmentation has been a long-standing challenge in the fields of computer vision and robotics over the past decade [3], [17]. SoTA networks generally fall into two groups [14]: (1) single-modal networks (with a single encoder) and (2) feature-fusion networks (with multiple encoders) [18]–[20]. Early efforts primarily focused on encoder-decoder architectures for pixel-level classification. Representative examples include PSPNet [5], the DeepLab series [6], [21], as well as Transformer-based networks [7], [22], [23]. The encoder extracts hierarchical, contextual feature maps from input images, while the decoder generates segmentation maps by upsampling and combining feature maps from different encoder layers. Nonetheless, these networks are limited in effectively combining heterogeneous features extracted from different sources (or modalities) of visual information, which makes it challenging to produce accurate segmentation results in scenarios with poor lighting and illumination conditions [18]. This has led researchers to focus on feature-fusion networks that can effectively fuse heterogeneous features learned from multiple sources (or modalities) of visual information. This problem is often referred to as “RGB-X semantic segmentation”, where “X” represents the additional modality (or source) of visual information, in addition to the RGB images. The most representative feature-fusion networks include convolutional neural network (CNN)-based ones, such as MFNet [24], RTFNet [25], SNE-RoadSeg series [18], [26], [27], and S\(^3\)M-Net [14], as well as Transformer-based ones, such as OFF-Net [28] and RoadFormer [29]. In this article, we design TiCoSS based on our previously proposed S\(^3\)M-Net, with a specific emphasis on exploring more effective solutions for tighter coupling between semantic segmentation and stereo matching.

2.2 Stereo Matching↩︎

Owing to recent advancements in deep learning techniques, end-to-end deep stereo networks [8], [9], [30]–[33] have dramatically outperformed traditional explicit programming-based stereo matching algorithms. PSMNet [8] uses a spatial pyramid to capture multi-scale information and uses multiple 3D convolutional layers to exploit both local and global contexts for cost computation. GwcNet [30] builds on PSMNet by constructing the cost volume through group-wise correlation, enhancing the 3D stacked hourglass network. To address the computational demands of 3D convolutions, researchers have sought ways to balance efficiency and accuracy in stereo matching. LEA-Stereo [32], for instance, introduces the first neural architecture search (NAS) [34] framework to stereo matching. RAFT-Stereo [9], a rectified stereo matching approach, developed based on RAFT [35], uses a series of gateD recurrent units to refine correlation features and enhance disparity estimation accuracy. CRE-Stereo [33] leverages recurrent refinement to update disparities in a coarse-to-fine fashion and adaptive group correlation to minimize the impact of erroneous rectification, leading to more compelling disparity estimation results. In this article, we primarily focus on improving semantic segmentation performance, and thus, adopt the stereo matching approach used in our previous work, S\(^3\)M-Net.

2.3 Simultaneous Semantic Segmentation and Stereo Matching↩︎

Existing joint learning frameworks that simultaneously address these two tasks mainly focus on improving disparity accuracy by leveraging semantic information [10], [12], [36]–[39]. However, discussions on utilizing disparity information to enhance semantic segmentation at the feature level for joint learning remain limited, except for the aforementioned “RGB-X semantic segmentation”. These prior arts often require extensive annotated training data or involve complex training strategies for joint learning. For example, SegStereo [36] and DispSegNet [37] necessitate an initial unsupervised training phase on the large-scale Cityscapes [40] dataset, followed by a subsequent supervised fine-tuning on the smaller KITTI [16], [41] datasets. Similarly, the approaches introduced in [12], [38], [39] require pre-training their spatial branches for stereo matching on the large-scale SceneFlow [42] dataset before fine-tuning both semantic and spatial branches on the KITTI [16], [41] datasets. DSNet [10] adopts a different joint learning strategy by alternating the training of the semantic segmentation and stereo matching networks, with each network being frozen during the training of the other. Nevertheless, achieving simultaneous convergence of the two networks can be challenging, as shared features are not learned in an end-to-end fashion. Recently, SSNet [43] uses a single encoder to extract shareable features for both tasks. However, as demonstrated in [44], such features may not be suitable for both dense prediction and geometric vision tasks. S\(^3\)M-Net uses two separate encoding branches to achieve these two tasks simultaneously, but the weak coupling between these branches limits the integration of contextual and geometric information. Additionally, the source code (in PyTorch or TensorFlow) for these previous approaches is not publicly available, and re-implementing them carries the risk of introducing errors. In contrast to the aforementioned approaches, our proposed TiCoSS uses a tightly-coupled joint learning framework that effectively leverages both contextual and geometric information. Moreover, TiCoSS is trained in an end-to-end fashion and is capable of learning semantic segmentation and stereo matching simultaneously, even with limited training data.

3 Methodology↩︎

3.1 Framework Overview↩︎

The architecture of our proposed TiCoSS is illustrated in Fig. 1, containing three major technical contributions:

1. A novel duplex, tightly-coupled encoder designed to selectively extract and fuse heterogeneous features, namely contextual features from RGB images and geometric features from disparity maps.

2. A novel HDS strategy that leverages fused features with the richest local spatial details to guide deep supervision across each branch (auxiliary classifier).

3. A CT loss that supervises the entire joint learning process and further tightens the coupling between semantic segmentation and stereo matching.

3.2 Tightly-coupled Gated Fusion Strategy↩︎

S\(^3\)M-Net [14] proposes an effective joint learning framework to simultaneously perform semantic segmentation and stereo matching. Despite achieving impressive results, these two tasks are loosely coupled. It merely employs the feature fusion strategy proposed in SNE-RoadSeg [18], where the geometric features extracted from disparity maps are indiscriminately fused into the contextual features extracted from RGB images via simplistic element-wise summation. The fused heterogeneous features are then treated as preceding contextual features and fed into subsequent layers without selective processing, which misleads the semantic segmentation task. This is primarily because the deeper geometric features contain irrelevant semantic information, and as the network goes deeper, the proportion of RGB features in the decoder’s input tends to diminish [27].

Our TGF strategy is, therefore, designed to overcome this limitation by selectively complementing contextual features with useful geometric features, resulting in a tightly-coupled duplex encoder. The core of our TGF strategy is the SIGs, developed based on Gated Fully Fusion (GFF) [45], which fuses features from multiple scales using gates that control the propagation of useful information. This enables the features at each scale to be enhanced by both deeper, semantically stronger features and shallower, spatially richer features, significantly reducing noises during feature fusion. Nonetheless, GFF is primarily regarded as a post-processing technique (multi-scale features are required to be yielded before being processed by GFF) and is infeasible to fuse heterogeneous features, which are extracted and fused progressively. In contrast, our developed SIGs (see Fig. 1) selectively inherit useful features in \(\mathbf{X}_{i-1}^{G,F}\) from the previous layer into \(\mathbf{X}_{i}^{G,F}\), the features at the current layer, where \(i\in[1,n]\cap\mathbb{Z}\) denotes the layer number, and the superscripts ‘\(G\)’ as well as ‘\(F\)’ represent ‘geometric’ and ‘fused’ features³, respectively. Our SIG outputs \(\mathbf{\tilde{X}}_{i}^{G,F}\), selectively inherited feature maps at the \(i\)-th layer using the following expression: \[\begin{align} \mathbf{\tilde{X}}_{i}^{G,F} &=\Omega_{i} \left( \mathbf{X}_{i-1}^{G,F}, \mathbf{X}_{i}^{G,F} \right) = \left(\mathbf{1}_{H_i}(\mathbf{1}_{W_i})^\top+\mathbf{G}_{i}\right) \odot \mathbf{X}_{i}^{G,F} \nonumber \\ &+ \left(\mathbf{1}_{H_i}(\mathbf{1}_{W_i})^\top-\mathbf{G}_{i}\right) \odot \left [ \mathbf{G}_{i-1} \odot \mathcal{R}(\mathbf{X}_{i-1}^{G,F}) \right], \end{align}\] where \(\Omega_i\) represents the SIG operation at the \(i\)-th layer, \(\mathbf{1}_k\) denotes a column vector of ones, \(\mathbf{G}_i \in [0,1]^{H_i \times W_i}\) represents a gate map that controls feature propagation, \(\odot\) denotes the element-wise multiplication broadcasting in the channel dimension, and \(\mathcal{R}\) represents the remapping operation, as detailed in [14].

Based on our proposed TGF strategy, the heterogeneous feature extraction and fusion process in our duplex encoder can be formulated as follows: \[\begin{align} &\mathbf{F}_{i}^{G}=\left\{ \begin{array}{ll} \mathcal{E}_{i}^{G}(\mathbf{D}^L), & i=1\\ & \\ \Omega_{i}^{G}(\mathbf{F}_{i-1}^{G}\,,\,\mathcal{E}_i^G(\mathbf{F}_{i-1}^G)), & 1<i\le n \end{array} \right. \\ \text{and} \notag \\ &\mathbf{F}_{i}^{F}=\left\{ \begin{array}{ll} \mathcal{E}_{i}^{F}(\mathbf{F}_i^C)\:\oplus \:\mathbf{F}_i^G, & i=1\\ & \\ \Omega_{i}^{F}(\mathbf{F}_{i-1}^{F}\,,\,\mathcal{R}(\mathbf{F}_{2i-1}^C))\:\oplus\: \mathbf{F}_i^G, & 1<i\le \frac{n+1}{2} \\ & \\ \Omega_{i}^{F}(\mathbf{F}_{i-1}^{F}\,,\,\mathcal{E}_i^F(\mathbf{F}_{i-1}^F))\:\oplus\: \mathbf{F}_i^G, & \frac{n+1}{2}<i\le n \end{array}, \right. \end{align}\] where \(\mathbf{D}^L\) denotes the estimated disparity map, \(\mathbf{F}^{C}_i\), \(\mathbf{F}^{G}_i\), and \(\mathbf{F}^{F}_i\) respectively represent the extracted contextual feature maps in the left view, geometric feature maps, and fused feature maps at the \(i\)-th layer, \(\mathcal{E}^G_i\) and \(\mathcal{E}^F_i\) denote the geometric and fused features encoding operations at the \(i\)-th layer, respectively, \(\oplus\) represents the element-wise summation. Considering that the shallower features between the two tasks have similar numbers of channels, we make the contextual feature maps of the first three layers share weights with the feature maps extracted from the stereo matching network in our practical implementation. Our proposed TGF strategy selectively propagates features to subsequent layers, reducing indiscriminate heterogeneous feature fusion which can severely mislead semantic segmentation as the network goes deeper, thus achieving tighter coupling between these two relevant environmental perception tasks.

3.3 Hierarchical Deep Supervision↩︎

After tightening the coupling between semantic segmentation and stereo matching during the feature encoding stage, we turn our focus towards the feature decoding aspect. We first revisit the deep supervision strategies employed in SNE-RoadSeg+ [26] and UNet 3+ [46]. The former applies deep supervision to the decoded features with the highest resolution, whereas the latter achieves this on the deepest decoded features at each resolution. Despite the effectiveness of these two prior approaches in addressing challenges such as vanishing gradients and slow model convergence, the deep supervision strategies employed by them are not comprehensive (the former emphasizes enhancing local details, while the latter focuses on improving multi-scale feature consistency). Ideally, they should be used in conjunction to complement each other for improved results. Therefore, our proposed HDS strategy combines the strengths of both methods and demonstrates superior performance compared to each individually.

A straightforward way to combine the strengths of these two methods is to apply deep supervision simultaneously to both the main and side branches, enabling the network to leverage features from shallow layers (containing rich local details) and deep layers (being semantically strong). However, this strategy does not fully exploit the potential complementarity between the main and side auxiliary classifiers, since the decoded features are exclusively derived from adjacently-connected ones. To improve their interactions, we utilize the decoded, fused feature maps \(\mathbf{F}_{1}^{F}\) (containing rich, fine-grained local spatial details that are essential for semantic segmentation) at the highest resolution to guide the feature decoding process in the side branches. Specifically, for the \(l\)-th auxiliary classifier within the side branch, we utilize a Feature Dynamic Alignment (FDA) block, which is composed of \(l\) downsampling units to progressively align channel dimensions and spatial resolutions between a pair of features at different layers. Each downsampling unit comprises a \(3\times3\) convolutional layer with a stride of 2, followed by a batch normalization (BN) layer and a rectified linear unit (ReLU) activation layer. Compared to the simple max-pooling operation which may disrupt the original feature representation, our FDA achieves a smoother feature alignment. The feature maps obtained by downsampling \(\mathbf{F}_{1}^{F}\) are then concatenated with the deepest decoded features at the corresponding layers. This downsampling output not only guides the decoding process but also serves as the input for the subsequent downsampling unit, thereby preserving fine-grained local details to the greatest extent possible. Since the outputs of the side branches are obtained by directly upsampling the deepest features at each layer, this strategy does not significantly impact training efficiency and memory usage. Moreover, the auxiliary classifiers of the main and side branches collaboratively provide additional pathways for gradients to propagate more efficiently from the output layers to their corresponding layers, thereby accelerating the convergence of our model.

3.4 Coupling Tightening Loss for Multi-task Joint Learning↩︎

Compared to the tasks focused solely on either semantic segmentation or stereo matching, the loss function employed in our joint learning framework aims to supervise both tasks simultaneously [14]. Our proposed CT loss function is expressed as follows: \[\mathcal{L}_{CT} = \mathcal{L}_{DIA} + \mathcal{L}_{DSCC}+\mathcal{L}_{SCG}+\mathcal{L}_{SM}, \label{eq46l95ct}\tag{1}\] where the DIA loss \(\mathcal{L}_{DIA}\) measures disparity inconsistency, the DSCC loss \(\mathcal{L}_{DSCC}\) measures the segmentation consistency across outputs from all deep supervision branches, the SCG loss \(\mathcal{L}_{SCG}\) denotes the Semantic Consistency-Guided (SCG) loss proposed in [14], and \(\mathcal{L}_{SM}\), a prevalently used loss function to supervise the training of the stereo matching network, is detailed in [14].

3.4.1 Disparity Inconsistency-Aware Loss↩︎

In our previous work [14], the developed SCG loss focuses mainly on regions with semantic inconsistencies caused by occlusion and reflection. Given that relying solely on semantic consistency cannot sufficiently tighten the coupling between these two tasks at the output level, we define a weight matrix \(\mathbf{W} \in \mathbb{R} ^{H\times W}\) drawing from the concept of Left-Right Consistency in stereo matching [11], where \[\mathbf{W}(\mathbf{p})=\, \mathbf{D}^L(\mathbf{p})-\mathbf{D}^R(\mathbf{p}-(\mathbf{D}^L(\mathbf{p};0))) \label{eq46weight95matrix}\tag{2}\] denotes the weight at the given pixel \(\mathbf{p}\), and \(\mathbf{D}^L\) as well as \(\mathbf{D}^R\) represent the left and right disparity maps, respectively. \(\mathbf{W}^{N}\in \mathbb{R} ^{H\times W}\), a normalized weight matrix, is then yielded, where \[\mathbf{W}^{N}(\mathbf{p})= \frac{1}{1+e^{-\left | \mathbf{W}(\mathbf{p}) \right |}}\in[0,1]. \label{eq46w95n}\tag{3}\] A higher normalized weight corresponds to a greater inconsistency between a given pair of left and right disparities, indicating the need for more attention during the training of a semantic segmentation model. Our proposed DIA loss is, therefore, formulated as follows: \[\mathcal{L}_{DIA}=\alpha\sum_{i=1}^{n} \left \{ -\frac{1}{N}\sum_{j=1}^{N}\sum_{k=1}^{C}\left [ \mathbf{W}^N(\mathbf{p})\,y_k(\boldsymbol{p})\,\mathrm{log}(\hat{y}_k(\boldsymbol{p}) ) \right ] \right \}, \label{eq46dia}\tag{4}\] where \(n\) denotes the number of deep supervision branches, \(N\) represents the total number of pixels, \(C\) denotes the total number of classes, \(\hat{y}_{k}(\boldsymbol{p})\) represents the predicted probability of pixel \(\boldsymbol{p}\) belonging to class \(k\), \(y_{k}(\boldsymbol{p})\) denotes the ground-truth label for \(\boldsymbol{p}\) in class \(k\), and the weight factor \(\alpha\) is determined through extensive ablation studies, as detailed in Sect. [sec:sec46exp95ablation].

3.4.2 Deep Supervision Consistency Constraint Loss↩︎

In prior research [26], [46], the relationship among prediction maps generated by different auxiliary classifiers was not considered, which may lead to semantic inconsistencies across scales. To address this issue, we draw inspiration from [47] to design the following DSCC loss, which utilizes the KL divergence to measure the prediction differences across scales: \[\mathcal{L}_{DSCC} = \beta \sum_{r=1}^{L} \sum_{\substack{s=1 \\ s \neq r}}^{L}\left [ -\frac{1}{N}\sum_{i=1}^{N}\hat{y}_k^r(\boldsymbol{p}) \log\frac{\hat{y}_k^r(\boldsymbol{p})}{\hat{y}_k^s(\boldsymbol{p})} \right ] , \label{eq46dscc}\tag{5}\] where l denotes the classifier index and \(\beta\) is a scaling factor determined through extensive ablation studies presented in Sect. [sec:sec46exp95ablation]. By minimizing the DSCC loss, we enhance the informativeness of features at different resolutions, thereby significantly improving semantic segmentation performance across scales.

4 Experiments↩︎

In this article, we conduct extensive experiments to evaluate the performance of our introduced TiCoSS both quantitatively and qualitatively. The following subsections detail the used datasets, experimental set-up, evaluation metrics, and the comprehensive evaluation of our proposed method.

4.1 Datasets↩︎

Since our network training requires both semantic and disparity annotations, we utilize the following two public datasets to conduct the experiments:

• The vKITTI2 [15] dataset contains virtual replicas (providing 15 semantic classes) of five sequences from the KITTI dataset. Dense ground-truth disparity maps are obtained through depth rendering using a virtual engine. Following [14], we employ 700 stereo image pairs, along with their semantic and disparity annotations, to evaluate the effectiveness and robustness of our proposed TiCoSS, where 500 pairs are utilized for model training and the remaining 200 pairs are used for model validation.

• The KITTI 2015 [16] dataset contains 400 stereo image pairs captured in real-world driving scenarios. Half of these pairs have ground truth annotations, while the remaining half do not. This dataset provides 19 semantic classes (consistent with those in the Cityscapes [40] dataset). Sparse ground-truth disparity maps are obtained using a Velodyne HDL-64E LiDAR. In our experiments, we split the data with a 7:3 ratio for training and testing, respectively.

4.2 Experimental Setup↩︎

Our experiments are conducted on an NVIDIA RTX 4090 GPU with a batch size of 1. We set the maximum disparity search range to 192 pixels. All images are cropped to 512 \(\times\) 256 pixels before being processed by the network. We utilize the AdamW [48] optimizer for model training, with epsilon and weight decay set to \(10^{-5}\) and \(10^{-8}\), respectively. The initial learning rate is set to \(2\times 10^{-4}\). Training lasts for 100,000 iterations on the vKITTI2 dataset and 20,000 iterations on the KITTI 2015 dataset. Standard data augmentation techniques are applied to improve model robustness.

4.3 Evaluation Metrics↩︎

Following [14], we quantify the performance of semantic segmentation using seven metrics: (1) accuracy (Acc), (2) mean accuracy (mAcc), (3) precision (Pre), (4) recall (Rec), (5) mean F1-score (mFSc), (6) mean intersection over union (mIoU), and (7) frequency-weighted intersection over union (fwIoU) [49]. Moreover, we quantify the performance of stereo matching using two metrics: (1) average end-point error (EPE) and (2) percentage of error pixels (PEP) at tolerance levels of 1.0 and 3.0 pixels, respectively.

1.5mm

1.5mm

1.5mm

4.4 Comparison with State-of-the-art Methods↩︎

4.4.1 Semantic Segmentation Performance↩︎

The qualitative experimental results on the KITTI and vKITTI2 datasets are presented in Figs. 2 and 3, respectively, while the quantitative experimental results on these two datasets are given in Tables 1 and 2, respectively.

These results suggest that TiCoSS outperforms all other SoTA single-modal and feature-fusion networks (including both CNN-based and Transformer-based methods) across all evaluation metrics on both datasets. Specifically, compared with S\(^3\)M-Net, the SoTA joint learning method, TiCoSS demonstrates substantial improvements on the KITTI dataset, achieving increases of \(9.68\%\) in mAcc, \(10.57\%\) in mIoU, \(2.71\%\) in fwIoU, and \(1.20\%\) in mFSc, respectively. Similarly, on the vKITTI2 dataset, it outperforms other networks across all evaluation metrics, with improvements of \(3.88\%\) in mAcc, \(5.08\%\) in mIoU, \(0.62\%\) in fwIoU, and \(0.26\%\) in mFSc, respectively. Particularly, as observed in Figs. 2 and 3, TiCoSS achieves more accurate predictions on distant regions as well as object boundaries and is capable of providing more fine-grained semantic segmentation details compared to S\(^3\)M-Net.

4.4.2 Stereo Macthing Performance↩︎

The qualitative experimental results on the KITTI and vKITTI2 datasets are illustrated in Figs. 5 and 6, respectively, while the quantitative experimental results on these two datasets are provided in Table 3. Since the primary focus of this study is to improve semantic segmentation performance, the stereo matching performance of TiCoSS is slightly better than that of S\(^3\)M-Net. Specifically, compared to S\(^3\)M-Net, TiCoSS demonstrates improvements of \(3.64\%\) in EPE and \(2.47\%\) in PEP 3.0 on the KITTI dataset. Additionally, on the vKITTI2 dataset, it achieves improvements of \(5.26\%\) in EPE and \(1.26\%\) in PEP 1.0. These improvements can be attributed to the tighter coupling between these two tasks, resulting in more comprehensive geometric features with informative contextual information compared to S\(^3\)M-Net. Additionally, by minimizing the CT loss, our model focuses more on areas with inconsistent disparities and achieves improved performance in occluded regions, as depicted in Fig. 6.

1.5mm

1.5mm

1.5mm

1.5mm

5 Conclusion and Future Work↩︎

This article introduced TiCoSS, a novel joint learning framework designed to tighten the coupling between semantic segmentation and stereo matching tasks. We have made three significant contributions in this work: (1) an effective feature fusion strategy that complements the contextual features with useful geometric features through a series of selective inheritance gates, enabling a tightly-coupled duplex encoder, (2) a novel hierarchical deep supervision strategy that improves the interactions among auxiliary classifiers, and (3) a coupling tightening joint learning loss that focuses on further tightening the coupling of these two tasks at the output level. The effectiveness of each contribution was validated through extensive experiments on both virtual and real-world datasets. Comprehensive comparisons with other state-of-the-art algorithms unequivocally demonstrate the superiority of TiCoSS.

References↩︎