1 Introduction↩︎

Stereo matching is a classical research topic in computer vision, which aims to estimate a disparity/depth map from a pair of rectified stereo images. It is a key enabling technique for various applications, such as autonomous driving [1], robot navigation [2], SLAM [3], [4], etc. Currently, impressive performances have been achieved by many deep learning-based stereo methods on most of the standard benchmarks.

However, significant domain shifts commonly exist among different datasets, which limits the generalization abilities of current state-of-the-art stereo matching methods. For example, the Middlebury [5] dataset mainly contains indoor high-resolution scenes while the KITTI dataset [6], [7] mainly consists of real-world urban driving scenarios. More specifically, as illustrated in Fig. 2 (a), there are significant differences among various datasets, e.g., indoors vs outdoors, color vs gray, and real vs synthetic. In addition, the disparity ranges are different among various datasets. As illustrated in Fig. 3, the disparity range of half-resolution images in Middlebury [5] is even more than 6 times larger than full-resolution images in ETH3D [8] (400 vs 64). Such unbalanced disparity distributions make the current approaches trained with a fixed disparity range difficult to cover the whole disparity range of another dataset without substantial finetuning.

Consequently, methods with state-of-the-art performance on one dataset often cannot achieve comparable results on other datasets without substantial adaptation. To relieve the problem, our conference paper CFNet[9] proposes a cascade and fused cost volume representation to narrow down the domain difference. By employing the cascade cost volume representation to alleviate the unbalanced disparity distribution, the method can eliminate the need for adaptation and performs well across a variety of datasets with fixed model parameters and hyperparameters, i.e., joint generalization. Unfortunately, such a method still needs suitable labeled target domain data, which cannot be easily obtained in most practical settings. Moreover, the labeled ground truth data is commonly obtained by expensive sensors (e.g. LiDAR) alongside careful calibration, which is cumbersome and costly, limiting the applicability in practical settings. Thus, we need to push methods to be robust and perform well across different datasets without using the groundtruth labels from the target domain.

In this paper, an Uncertainty-based Cascade and Fused cost volume representation (UCFNet) is proposed to dig into uncertainty estimation for robust stereo matching. Specifically, an uncertainty-based pseudo-labels generation method is proposed to adapt the pre-trained model to the new domain, i.e., domain adaptation. A key observation behind our method is that learning-based models can be successfully adapted to new domains even by deploying only a few sparse groundtruth annotations. For example, learning-based models can achieve state-of-the-art performance on KITTI datasets with limited sparse groundtruth (less than 1/3 pixels is annotated for totally 200 images). Thus, we can employ the proposed uncertainty estimation to filter out the high-uncertainty pixels of the pre-trained model and generate sparse while reliable disparity maps as pseudo-labels to adapt the pre-trained model. As shown in Fig. 1, the provided ground truth data is sparse and cannot provide valid annotation in the upper region of the scene. Instead, the proposed method can generate a denser disparity map as pseudo-labels, which can filter out most errors of UCFNet_pretrain and cover all regions of the input picture. Consequently, the proposed method can tremendously improve the performance of our pre-training model on textureless area of foreground (red dash boxes) and unlabeled area of background (green dash boxes) by solely employing self-generated proxy labels as ground truth. More specifically, pixel-level and area-level uncertainty estimation are employed to generate reliable pseudo-labels. Given current disparity estimation results, we first employ pixel-level uncertainty estimation to quantify the degree to which the current disparity probability distribution tends to be multi-modal and employ it to evaluate the pixel-level confidence of current estimations. Then, area-level uncertainty estimation is proposed to leverage the multi-modal input and neighboring pixel information to further refine the initial uncertainty map. By the cooperation between pixel-level and area-level uncertainty estimation, we can obtain a denser and more robust pseudo label for domain adaptation without requiring cumbersome and expensive depth annotations.

Experimentally, we perform extensive experimental evaluations on various benchmarks to verify the generalization of the proposed method. When trained on synthetic datasets and generalized to unseen real-world datasets, our pre-trained model shows strong cross-domain generalization and can generate a good initial value for subsequent adaptation. Then, our model can further promote its performance by solely feeding the target domain synchronized stereo images and generated pseudo-labels, i.e., without the need for ground truth. In specific, the proposed method outperforms other domain generalization/adaptation methods by a noteworthy margin on various stereo matching benchmarks. The Qualitative comparison among GANet_pretrain, UCFNet_pretrain, and UCFNet_adapt on three real datasets is shown in Fig. 2. It can be seen from the figure that the generalization of current dataset-specific methods is limited to unseen real scenes, while our pre-training method can correct most errors and generate a more reasonable result. Moreover, compared with the pre-training model UCFNet_pretraining, the proposed UCFNet_adapt can achieve consistent improvement on multiple datasets with different characteristics, which further verifies the effectiveness of the generated pseudo-labels. More visualization results can be seen in the video demo of supplementary. Additionally, our uncertainty-based pseudo-labels can further be extended to replace the ground truth of monocular depth estimation networks and train these networks in an unsupervised way. Experiments show that the deep monocular depth estimation network trained by our pseudo-labels can outperform all self- supervised monocular depth estimation algorithms by a noteworthy margin and even achieves comparable performance with supervised methods. The code will be available at https://github.com/gallenszl/UCFNet.

In summary, our main contributions are:

• We propose an uncertainty-based cascade and fused cost volume representation to reduce the domain differences and balance different disparity distributions across a variety of datasets. Thus, a robust pre-trained model with strong cross-domain generalization can be obtained.

• We propose an uncertainty-based pseudo-labels generation method to further narrow down the domain gap. By employing the generated pseudo-labels to adapt our pre-trained model to the new domain, we can greatly promote the performance of our method.

• Our method shows strong cross-domain and adapt generalization and outperforms other domain generalization/domain adaptation methods by a noteworthy margin on various stereo matching benchmarks.

• Our method can perform well on multiple datasets with fixed model parameters and hyperparameters and obtains 1st place on the stereo task of Robust Vision Challenge 2020 ¹.

• Our uncertainty-based pseudo-labels can further be extended to train monocular depth estimation networks in an unsupervised way and even achieves comparable performance with supervised methods.

Differences with conference version [9]: This paper extends the early ideas and findings presented in CFNet[9]. The differences with our conference paper can be summarized as follows:

• In our previous work, we only focus on cross-domain generalization and joint generalization of stereo matching tasks. Here, we provide a general solution for cross-domain, joint, and adaptation generalization jointly by digging into uncertainty estimation in stereo matching. Hence, a more complete and standard solution is presented for robust stereo matching.

• In our previous work, the final disparity estimation is just half-resolution of the input image and needs to be upsampled to the original image size. Thus, we propose a simple, yet effective attention-based refinement module to recover the details loss caused by the bilinear sampling.

• We extend our uncertainty-based pseudo-labels to train the monocular depth estimation network in an unsupervised way. Experiments show that the deep monocular depth estimation network trained by our pseudo-labels can outperform all self-supervised monocular depth estimation algorithms by a noteworthy margin and even achieves comparable performance with the supervised methods.

2 Related Work↩︎

2.1 Multi-scale Cost Volume based Stereo Matching↩︎

Cost volume construction is an indispensable step in the well-known four-step pipeline for stereo matching [10]–[12]. Typically, current state-of-the-art stereo matching methods can be categorized into two types of cost volume-based methods, where the cost volume is a 4D tensor of height, width, disparity, and features. The first category usually uses the single-feature 3D cost volume generated by full correlation, which is efficient while losing much information due to the decimation of feature channels. Many real-time methods, such as Dispnet [13], MADNet [14], [15] and AANet [16], belongs to the category. Moreover, two-stage refinement [17] and pyramidal towers [14] are commonly applied in the single-feature cost volume based network to construct multi-scale cost volume. The second category usually uses the multi-feature 4D cost volume generated by concatenation [18] or group-wise correlation [19], which can achieve better performance with higher computational complexity and memory consumption. Most top-performing networks, including GANet [20], CSPN [21] and ACFNet [22] belong to this category. Recently, to alleviate the high computational complexity and memory consumption when employing multi-feature 4D cost volumes, [23]–[25] propose to use cascade cost volume representation in multi-view stereo. These methods usually first predict an initial disparity at the coarsest resolution of the image and then gradually refine the disparity by narrowing down the disparity search space. More closely related to our approach is Casstereo [24], which first extended such representation to stereo matching. It selected to uniform sample a pre-defined range to generate the next stage’s disparity search range. Instead, we employ pixel-level uncertainty estimation to adaptively adjust the next stage disparity searching range and generate pseudo-labels for subsequent domain adaptation. Our method also shares similarities with UCSNet [25], which constructs uncertainty-aware cost volume in multi-view stereo while it doesn’t employ uncertainty estimation to generate pseudo-labels.

2.2 Robust Stereo Matching↩︎

There exist three categories of generalization definitions for robust stereo matching. 1) Cross-domain Generalization: the network’s ability to perform well on unseen scenes (cannot see the image pairs of the target domain in advance). Towards this end, Jia et al [26] propose to incorporate scene geometry priors into an end-to-end network. Zhang et al [27] introduce a domain normalization and a trainable non-local graph-based filter to construct a domain-invariant stereo matching network. 2) Adapt Generalization: the network’s ability to adapt pre-trained models to the new domain with unlabeled target data. Previous work usually pre-trains the models on synthetic data and then adapts it to new target domains with Graph Laplacian regularization [28], non-adversarial progressive color transfer [29], and Knowledge Reverse Distillation [30]. More closely related to our approach are [30], [31] in stereo matching and Monoresmatch [32] in monocular depth estimation, which also proposes to generate a pseudo-label for domain adaptation. However, these methods all select to employ classical stereo matching methods [33] alongside with confidence estimators, e.g., left-right consistency check to generate pseudo-labels. That is all these methods need an independent method to generate corresponding pseudo-labels. Instead, the proposed method is an end-to-end network that can generate the predicted disparity map, corresponding uncertainty map and pseudo-labels jointly, which is a more simple, yet efficient way. 3) Joint Generalization: the network’s ability to perform well on a variety of datasets with the same model parameters. MCV-MFC [17] introduces a two-stage finetuning scheme to achieve a good trade-off between generalization and fitting capability on multiple datasets. However, it doesn’t touch the inner difference between diverse datasets, e.g, the unbalanced disparity distribution. To further address this problem, we propose a cascade cost volume to adaptively the next stage disparity searching space, where the pixel-level uncertainty estimation is at the core.

3 Our Approach↩︎

3.1 Framework Overview↩︎

In this paper, we provide a general solution for cross-domain generalization, joint generalization, and adaptation generalization jointly by digging into uncertainty estimation in stereo matching. The overall architecture of our method is shown in Fig. 4, which can divide into three steps:

1) Training a robust stereo matching network on source domain: Given stereo image pairs and corresponding ground-truth disparity of source domain (synthetic dataset), we first propose to employ the synthetic data to train a robust pre-train model with strong cross-domain generalization. Specifically, an Uncertainty-based Cascade and Fused cost volume representation (UCFNet) is proposed to alleviate the unbalanced disparity distribution and large domain shifts across different datasets.

2) Pseudo-label generation on target domain: After getting the pre-training model, we propose an uncertainty-based pseudo-label generation method to generate reliable pseudo-labels for domain adaptation. Specifically, the proposed method can be divided into two steps: (a) Given stereo image pairs of the target domain (real dataset), the pre-trained model is employed to predict the corresponding disparity estimation. (b) Two terms of uncertainty estimation, i.e., pixel-level and area-level are proposed to filter out the high-uncertainty pixels of the disparity estimation and generate sparse while reliable disparity maps as pseudo-label.

3) Domain adaptation with generated Pseudo-label: After getting the generated pseudo-label, we can employ it as supervision to adapt the pre-train model to the target domain. In addition, the generated pseudo-label can also be employed as supervision to train the Monocular depth estimation network in an unsupervised way. Experiments show the superiority of the proposed method on both monocular and binocular depth estimation tasks.

The structure of this paper is organized as follows. In Sec. 3.2, we present the details about how to employ the Uncertainty-based Cascade and Fused cost volume representation (UCFNet) for robust disparity estimation. Sec. 3.3 introduces the design of uncertainty estimation, which can filter out unreliable points of current estimations and generate reliable and sparse pseudo-label for subsequent domain adaptation. Sec. 3.4 introduces the mechanism of domain adaptation, i.e., how to employ the generated pseudo-label adapting the binocular/monocular depth estimation network to the new domain. Finally, we evaluate the results of our algorithms on both stereo matching and monocular depth estimation tasks in Sec. 4 and Sec. 5, respectively.

3.2 Uncertainty based cascade and fused cost volume for disparity estimation↩︎

To achieve robust stereo matching, we propose an Uncertainty-based Cascade and Fused cost volume representation (UCFNet) to alleviate the unbalanced disparity distribution and large domain shifts across different datasets. As shown in Fig. 5 and 8, the proposed UCFNet consists of four parts, including pyramid feature extraction, fused cost volume, cascade cost volume, and attention-based disparity refinement.

3.2.1 Pyramid feature extraction↩︎

Given an image pair, an unet-like [34], [35] encoder-decoder architecture is first proposed to extract multi-scale image features. Specifically, the encoder consists of five residual blocks, followed by an SPP [36] module to better extract hierarchical context information. Compared with the widely used Resnet-like network [19], [24], experiments show that the proposed unet-like feature extraction can preserve sufficient information with lower computational complexity. Then, the extracted multi-scale features can be divided into fused and cascade cost volumes and predict corresponding resolution disparity, respectively.

3.2.2 Fused Cost Volume↩︎

In this section, multiple low-resolution dense cost volumes are fused together to reduce the domain shifts across different datasets for initial disparity estimation. Our method is motivated by a simple observation that multi-scale cost volume can cover multi-scale receptive fields and drive the network to extract multi-level information, e.g., edges and areas are easier to be captured by low-resolution cost volume. Moreover, edges and areas are no-local information, which is less sensitive to domain changes. Hence, we can fuse multiple low-resolution dense cost volumes to incorporate hierarchical structural representations and generate a more accurate initial disparity estimation. Specifically, we first employ the input multi-scale features (smaller than 1/4 of the original input image resolution) to construct each scale cost volume respectively and then design a cost volume fusion module to integrate them. Details of the two steps will be provided below.

Cost volume construction: Inspired by [19], [37], feature concatenation and group-wise correlation are employed to generate corresponding combination volume as follows: \[\begin{align} \begin{array}{c} V_{concat}^i({d^i},x,y,f) = f_L^i(x,y)||f_R^i(x - {d^i},y) \\ \\ V_{gwc}^i({d^i},x,y,g) = \frac{1}{{N_c^i/{N_g}}}\left\langle {f_l^{ig}(x,y),f_r^{ig}(x - {d^i},y)} \right\rangle \\ \\ V_{combine}^i = V_{concat}^i||V_{gwc}^i \end{array} \label{eq:volume32construction} \end{align}\tag{1}\] where \(||\) denotes the vector concatenation operation. \({N_c}\) represents the channels of extracted features. \({N_g}\) is the amount of group. \(\left\langle {{\rm{ }},{\rm{ }}} \right\rangle\) represents the inner product. \({f^i}\) denotes the extracted feature at scale (stage) \(i\) and \(i = 0\) represents the original input image resolution.

Note that the disparity searching index \({d^i}\) is defined as \({d^i} \in \{ 0,1,2 \ldots \frac{{{D_{\max }}}}{{{2^i}}} - 1\}\) and the hypothesis plane interval equals to 1 in the fused cost volume representation. That is, these cost volumes are all dense cost volumes with the size of \(\frac{H}{{{2^i}}} \times \frac{W}{{{2^i}}} \times \frac{{{D_{\max }}}}{{{2^i}}} \times F\). By densely sampling the whole disparity range in small resolution, we can efficiently generate the coarsest disparity map. Then pixel-level uncertainty estimation is employed to narrow down the disparity searching space at higher resolution and refine the disparity estimation in a coarse-to-fine manner. Please refer to Section 3.2.3 for more detail.

Cost Volume fusion: The architecture of cost volume fusion is shown in Fig. 6. Specifically, we first employ four 3D convolution layers with skip connections to regularize each cost volume. Then, a 3D convolution layer (stride of two) is employed to downsample the scale 3 cost volume from 1/8 to 1/16 of the original input image size. Next, we concatenate the down-sampled cost volume and the next scale combination volume at the feature dimension and use one additional 3D convolution layer to decrease the feature channel to a fixed size. Similar operations are progressively employed until we downsample the cost volume to 1/32 of the original input image size. Finally, a 3D transposed convolution is adopted to up-sample the volume in the decoder and one 3-D hourglass network is further employed to aggregate the cost volume. Moreover, an output module is applied to predict the disparity from the fused cost volume. Specifically, we first employ two more 3D convolution layers to obtain a 1-channel 4D volume. Then, soft argmin [18] operation is applied to transform volume into probability and generate the initial disparity map \({D^3}\). The soft argmin operation is defined as: \[\begin{align} \widehat {{d^i}}{\rm{ = }}\sum\limits_{d = 0}^{\frac{{{D_{\max }}}}{{{2^i}}} - 1} {d \times \sigma ( - c_d^i)}, \label{softmax} \end{align}\tag{2}\] where \(\sigma\) denotes the softmax operation and \(c\) represents the predicted 1-channel 4D volume. \({\sigma ( - c_d)}\) denotes the discrete disparity probability distribution and the estimated disparity map is susceptible to all disparity indexes.

3.2.3 Cascade Cost Volume↩︎

Given the initial disparity estimation, the next step is to construct a fine-grained cost volume and refine disparity maps in a coarse-to-fine manner. One naive way to construct the next stage disparity searching range is uniform sampling a pre-defined searching range [24]. However, such a method treats all pixels equally and cannot make pixel-level adjustments. Furthermore, the unbalanced disparity distribution across different datasets requests networks to adjust the disparity searching range according to the input image adaptively. Hence, a question arises, can we drive the network to filter out invalid disparity indexes in a large disparity searching range and capture more possible pixel-level disparity searching space with prior knowledge of the last stage’s disparity estimation?

To tackle this problem, we propose a pixel-level uncertainty estimation to adaptively adjust the disparity searching range. As mentioned in Eq. 2 , the final predicted disparity can be obtained by softly weighting indices according to their probability. Thus, the discrete disparity probability distribution indeed reflects the similarities between candidate matching pixel pairs and the ideal disparity probability distribution should be unimodal peaked at true disparities. However, the actual probability distribution is predominantly unimodal or even multi-modal at some pixels, e.g., ill-posed and occluded areas. Moreover, existing methods [18], [22] have observed that the degree of multimodal distribution is highly correlated with the probability of prediction error. Hence, we propose to define a pixel-level uncertainty estimation to quantify the degree to which the cost volume tends to be multi-modal distribution and employ it to evaluate the confidence of the current estimation. The pixel-level uncertainty is defined as: \[\begin{align} \begin{array}{c} {{\rm{U}}^i}{\rm{ = }}\sum\limits_{\forall {d^i}} {{{(d - {{\hat{d}}^i})}^2} \times \sigma ( - c_d^i)} \\ \\ \widehat {{d^i}}{\rm{ = }}\sum\limits_{\forall {d^i}} {d \times \sigma ( - c_d^i)} \end{array} \end{align}\] where \(\sigma\) denotes the softmax operation and \(c\) represents the predicted 1-channel 4D volume. Fig. 7 gives a toy sample to show the effectiveness of pixel-level uncertainty estimation. As shown, the uncertainty of unimodal distribution equals to 0 and the more the distribution tends to be multimodal, the higher the error and uncertainty. Thus, we can employ pixel-level uncertainty to evaluate the confidence of disparity estimation, higher uncertainty implies a higher probability of prediction error and a wider disparity searching space to correct the wrong estimation. Then, the next stage’s disparity searching range can be defined as: \[\begin{align} \begin{array}{l} d_{\max }^{i - 1} = \delta (\widehat {{d^i}} + \left( {{\alpha ^i} + 1} \right)\sqrt {{U^i}} + {\beta ^i})\\ \\ d_{\min }^{i - 1} = \delta (\widehat {{d^i}} - \left( {{\alpha ^i} + 1} \right)\sqrt {{U^i}} - {\beta ^i}) \end{array} \end{align}\] where \(\delta\) denotes bilinear interpolation. \(\alpha\) and \(\beta\) are normalization factors, which are initialized as 0 and gradually learn a weight. Then, uniform sampling can be employed to get the next stage discrete hypothesis disparity indexes \({d^{i - 1}}\): \[\begin{align} \begin{array}{c} {d^{i - 1}} = d_{\min }^{i - 1} + n(d_{\max }^{i - 1} - d_{\min }^{i - 1})/\left( {{N^{i - 1}} - 1} \right)\\ \\ n \in \{ 0,1,2 \ldots {N^{i-1}} - 1\} \end{array} \end{align}\] where \({N^{i - 1}}\) is the number of hypothesis planes at stage \(i-1\). Then, a sparse while fine-grained cost volume at stage \(i-1\) (\(\frac{H}{{{2^{i - 1}}}} \times \frac{W}{{{2^{i - 1}}}} \times {N^{i - 1}} \times F\)) can be constructed based on Eq.@eq:eq:volume32construction . After getting the next stage cost volume, a similar cost aggregation network (omitting the solid line in Fig. 6) can be employed to predict the corresponding stage disparity map. By iteratively narrowing down the disparity range and higher the cost volume resolution, we can refine the disparity in a coarser to fine manner. Note that the final output of cascade cost volume \(\hat{d^{1}}\) is half resolution of the original image. Thus, an up-sampling operation is necessary to up-sample \(\hat{d^{1}}\) to the same size of original images, i.e., \(\hat{d^{0}} = up(\hat{d^{1}})\), where the up-sampling operation up is implemented by bilinear interpolation.

In summary, the proposed UCFNet outperforms previous cascade-based approaches, i.e., casstereo [24] in the following three aspects: First, we propose to fuse multiple dense low-resolution cost volumes to generate a more accurate initial disparity estimation at lower resolution (see the comparison between the estimation of casstereo at stage 2 and our UCFNet at stage 3 in Fig. [fig:32cost32volume32distribution] (a) and (b)). Second, pixel-level uncertainty estimation is proposed to adaptively adjust the next stage disparity searching range which can push disparity distribution to be more predominantly unimodal (Fig. [fig:32cost32volume32distribution](b)). Third, our method can better cover the corresponding ground truth value in the final stage disparity searching range by the proposed Uncertainty-based Cascade and Fused cost volume representation and corrects some biased results in casstereo (Fig. [fig:32cost32volume32distribution](a)).

3.2.4 Attention-based disparity refinement↩︎

As mentioned in Section 3.2.3, the output of our cascade cost volume \(\hat{d^{0}}\) is up-sampled from the half-resolution disparity map \(\hat{d^{1}}\) by bilinear interpolation. However, such direct upsampling operations will lead to the degradation of texture information, which indeed hinders both the finetuning performance and generalization of the proposed method. Hence, we propose a lightweight attention-based disparity refinement module to make up for the missing details. Experiments in Tab.1&2&3 demonstrate the proposed refinement network can achieve consistent improvement in both generalization and finetuning performance across multiple datasets.

The pipeline of the attention mechanism is shown in Fig. 8. Taking up-sampled disparity \(\hat{d^{0}}\) as input, we first employ multiple stacked convolution layers to extract the deep feature representation \(f_{input}\). Then, three encoding operations P(.), Q(.) and V(.) are used to convert \(f_{input}\) to three components \({f_p}\), \({f_Q}\) and \({f_V}\), in which reshape operation is utilized to convert the shape of \({f_Q}\), \({f_V}\) to \({f_Q} \in [C \times HW]\) and \({f_V} \in [HW \times C]\): \[\begin{align} \begin{array}{l} {f_{input}} = Conv({\hat{d^{0}}}),\\ {f_p} = P({f_{input}}),\\ {f_Q} = reshape(Q({f_{input}})),\\ {f_V} = reshape(V({f_{input}})), \end{array} \end{align}\] where Conv means convolution operation. Then, a matrix multiplication \(\otimes\) and a softmax operation are introduced to generate the attention weight, which reflects the similarity between each channel position of input feature map \(f_{input}\). Next, we employ a matrix multiplication operation between weights and \(f_p\) with a 2D convolution layer to generate the residual disparity: \[\begin{align} \hat{d_{residual}^0} = Conv(Weight \otimes {f_p} + {f_{input}}), \end{align}\] where Conv means convolution operation. Finally, an element-wise addition operation is employed to generate the refined disparity: \[\begin{align} \hat{d_{refine}^0} = \hat{d_{residual}^0} + \hat{{d^0}} \end{align}\]

Visualization results are shown in Fig. 9. All methods are only trained on the Scene Flow datatest and tested on unseen KITTI datasets. As shown, the original disparity estimation results lack texture information, e.g., the missing thin structures (see red dash boxes in the picture) and unsmooth regions due to the segmentation of lane lines (see yellow dash boxes in the picture). Instead, the proposed attention-based disparity refinement module can learn such missing details by the residual disparity (sub-figure (b)) and generate a better refined disparity map.

3.2.5 Loss function↩︎

Inspired by previous work [36], we employ smooth \(L_1\) loss function [38] to train the proposed stereo matching network. Specifically, the loss function is described as: \[\begin{align} L\left( {\hat{D},{D_{gt}}} \right) = \frac{1}{{{N_{gt}}}}\sum\limits_{i = 1}^{N_{gt}} {smoot{h_{{L_1}}}({D_{gt}} - \hat{D})} \end{align}\] in which \[\begin{align} Smoot{h_{{L_1}}}(x) = \left\{ \begin{array}{l} 0.5{x^2},{\rm{ if }}\left| x \right| < 1\\ \left| x \right| - 0.5,{\rm{ otherwise}} \end{array} \right. \end{align}\] where \(N_{gt}\) denotes the number of available pixels in the provided ground truth disparity of source domain and \(\hat{D}\) represents the predicted disparity.

3.3 Uncertainty estimation for pseudo-label generation↩︎

We propose an uncertainty based pseudo-label generation method to generate low-noise disparity maps and leverage them as supervision to adapt the pre-trained model to the target domain. A key observation behind our method is that deep stereo matching methods can be successfully adapted to a new domain by only deploying sparse ground-truth labels or even sparse noisy predictions [31], [39]. Based on the above observations, we propose to employ the target domain image pairs \(({I_L},{I_R})\) and the pre-trained model \({M_{{\rm{pre}}}}\) to generate dense disparity maps \({D_{{\rm{pre}}}}\). Then we can leverage the uncertainty estimation to filter out unreliable points of \({D_{{\rm{pre}}}}\) and generate reliable and sparse disparity maps \({D_u}\). Specifically, two terms of uncertainty estimation, i.e., pixel-level and area-level are employed to generate low-noise disparity maps. Below we will introduce each term of uncertainty estimation for more details.

Pixel-level Uncertainty Estimation: As mentioned in Sec. 3.2.3, we propose a pixel-level uncertainty estimation to adaptively adjust the next stage disparity searching range. As mentioned before, the proposed pixel-level uncertainty estimation is according to the sharpness of cost volume distribution to evaluate the confidence of current estimations. That is the proposed pixel-level uncertainty estimation has no learnable parameters and is totally decided by the input cost volume distribution. As the geometry of cost volume distribution is domain-invariant, we can ensure the generalization ability of pixel-level uncertainty estimation. Hence, intuitively, we can directly introduce the pixel-level uncertainty estimation to generate the corresponding pseudo-label \({D_{pixel}}\) as follows: \[\begin{align} \begin{array}{c} {D_{{\rm{pre}}}} = {M_{{\rm{pre}}}}({I_L},{I_R}),\\ \\ {D_{pixel}} = \{ d \in {D_{pre}}:\sqrt U < t\}, \end{array} \end{align}\] where \(t\) is the threshold that controls the density and reliability of the filtered disparity map \({D_{pixel}}\). A lower value of t will filter out more mistakes in \({D_{{\rm{pre}}}}\) and generating more sparse disparity maps \({D_{pixel}}\). Thus, by setting a reasonable threshold, we can utilize filtered disparity maps \({D_u}\) as if they were ground truth to supervise the fine-tuning of the pre-trained model \({M_{{\rm{pre}}}}\). As shown in Fig. 12, the proposed pixel-level uncertainty estimation can filet out most error of the cross-domain disparity estimation \({D_{{\rm{pre}}}}\) and generate low-noise disparity maps \({D_{pixel}}\). However, the proposed pixel-level uncertainty estimation still has the following two shortcomings: 1) it only evaluates pixel-level confidence and doesn’t consider the influence of neighboring pixels or global information. 2) it only employs the cost volume as input curs while ignoring the usage of multi-input, i.e., reference image and disparity maps. Thus, area-level uncertainty estimation is essential for the further refinement of pixel-level uncertainty maps.

Area-level Uncertainty Estimation: We propose an area-level uncertainty estimation to leverage the information of multi-modal input and neighboring pixels. Intuitively, both neighborhood and multi-modal information can better guide the pixel-level uncertainty map to identify the prediction correct region. For example, assuming that a pixel has high pixel-level uncertainty while the pixel-level uncertainty of surrounding similar pixels is relatively low, then we can judge it is the correct prediction to preserve more valid labels. In this case, the neighborhood information can guide the network to discover surrounding pixels and the multi-modal information can help the network to distinguish whether these pixels are similar and can be considered to have close uncertainty. Specifically, let us denote the input initial uncertainty map as \({U_{pixel}}\). Our goal is to recover from \({U_{pixel}}\) an improved uncertainty map \({U_{area}}\) that can more accurately identify the region where the disparity estimation is wrong. Such a task can be seen as a binary classification mission, where the output of the area-level uncertainty estimation will be constrained in the range of \((0,1)\). A higher value denotes a higher possibility of prediction error, i.e., higher uncertainty. Actually, such a task is very similar to the setting of salient object detection, which also employs a binary classification network to identify the visually distinctive regions or objects in a scene. Moreover, the usage of multi-modal input and neighboring information is an essential topic in salient object detection and has drawn great attention from the community. Hence, we propose to employ some network design ideas in salient object detection [40]–[43] to construct our area-level uncertainty estimation network. The overall architecture of our area-level uncertainty estimation network is shown in Fig. 10, which consists of six parts: left image encoder blocks \(E_l^i\), uncertainty map encoder blocks \(E_U^i\), fusion blocks \(F^i\), gate unit \(G^i\), main decoder block \(D_m^i\) and residual decoder block \(D_r^i\) (\(i\) denotes different scales and \(i = 0\) represents the original input image resolution). Below we will introduce each part in more detail.

Encoder blocks: Our encoder blocks can be divided into two parts: the reference image encoder block \(E_l^i\) and uncertainty map encoder block \(E_U^i\), which characterizes the information of the reference image and the concatenation of pixel-level uncertainty map and predicted disparity map, respectively. Specifically, we propose to employ the commonly used pre-trained backbone network ResNet-34 to construct our encoder block. Similar to previous work, we remove the last fully-connected and pooling layers of the employed backbone network.

Fusion blocks: Our fusion block has two main inputs: 1) left image encoder blocks, which represent the multi-scale information of reference images. 2) uncertainty map encoder block, which stores the multi-scale information of pixel-level uncertainty map and predicted disparity map. By employing the fusion blocks, we can integrate scale-matching reference image encoder blocks and uncertainty map encoder blocks to extract robust multi-modal input representation. Specifically, the fusion process can be formulated as: \[{F^i} = \delta({ E_l^i }||{ E_U^i }), \label{eq:decoder32block}\tag{3}\] where \(||\) denotes the concatenation operation and \(\delta\) refers to the convolution layer.

Gate unit based decoder block: Inspired by previous work [42], [44], we propose the gate unit based decoder block to control the message passing between scale-matching fusion and decoder blocks. In specific, our gate unit is formulated as:

where \(AvgPool\) denotes the global average pooling, \(S\) is the sigmoid function, \(||\) represents the concatenation operation, \(\delta\) refers to the convolution layer and \(FASPP\) is the Fold-ASPP operation [43]. Note that the output channel of \(\delta\) is 2. Hence, the proposed gate unit has two outputs, i.e., \(G_m^i\) and \(G_r^i\) , which will be employed to control the message passing in main decoder block \(D_m^i\) and residual decoder block \(D_r^i\), respectively. Specifically, \(G_m^i\) is employed to balance the contribution between upsampled main decoder blocks \(D_m^i\) and corresponding fusion blocks \(F^i\). The whole process can be written as: \[D_m^i = \left\{ \begin{array}{l} \delta (G_m^i \times \delta ({F^i}) + Up(D_m^{i + 1})){\rm{ if }}\;i = 1,2,3,4\\ \delta (G_m^i \times FASPP({F^i})){\rm{ if}}\; i = 5 \end{array} \right.\] where \(\times\) is the element-wise multiplication operation and \(Up\) refers to the up-sampling operation which is implemented by bilinear interpolation. Then we further introduce the residual decoder block \(D_r^i\) to recover the missed details of the main decoder blocks and also employ the gate unit \(G_r^i\) to balance the information flow. The whole process can be formulated as: \[D_r^i = \left\{ \begin{array}{l} G_r^i \times \delta ({F^i})||Up(D_r^{i - 1}{\rm{) if }}\;i = 1,2,3,4\\ G_r^i \times \delta ({F^i}){\rm{ if}}\; i = 5 \end{array} \right.\] Finally, we can fuse the output of two terms of decoder blocks to generate the area-level uncertainty map \({U_{area}}\): \[{U_{area}} = S(\delta (D_r^1||D_m^1) + D_m^1)\] where \(S\) is the sigmoid function. Thus, the output of the area-level uncertainty estimation \({U_{area}}\) is in the range of \((0,1)\) and a higher value denotes a higher possibility of prediction error. Then, we can use the same operation introduced in pixel-level uncertainty estimation to generate the corresponding pseudo-label \({D_{area}}\) as follows: \[\begin{align} \begin{array}{c} {D_{area}} = \{ d \in {D_{pre}}:{U_{area}} < t\} \end{array} \end{align}\] where \(t\) is the threshold that controls the density and reliability of the filtered disparity map \({D_{area}}\).

3.3.1 Loss function↩︎

We employ the cross-entropy loss to train the proposed area-level uncertainty estimation network. Specifically, the cross-entropy loss can be defined as: \[\begin{align} l = U_{gt}\log {U_{area}} + (1 - U_{gt})\log (1 - {U_{area}}) \end{align}\] where \(U_{area}\) and \(U_{gt}\) denote the predicted area-level uncertainty map and ground truth uncertainty mask, respectively. As the ground truth uncertainty mask is not provided in the source domain(synthetic dataset) and indeed the value of it will change as the convergence of the stereo matching network. Here, we specify how to obtain \(U_{gt}\) on source domain according to the provided ground truth disparity \(D_{gt}\) and predicted disparity \(\hat{D}\). Specifically, our ground truth uncertainty mask is defined as: \[\begin{align} {U_{gt}} = \left\{ \begin{array}{l} 1,{\rm{ if }}\left| {{D_{gt}} - \hat{D}} \right|{\rm{ > }}\delta {\rm{ }}\\ 0,{\rm{ otherwise}} \end{array} \right. \end{align}\] where \(\delta\) is the threshold that controls the strictness of uncertainty estimation, e.g., a higher \(\delta\) denotes a larger gap between \(D_{gt}\) and \(\hat{D}\) can be seen as a correct prediction.

3.4 Domain Adaptation with supervision of pseudo-label↩︎

After getting the generated pseudo-label \({D_{area}}\), we can employ it to adapt the pre-trained binocular depth estimation network to the new domain. The loss function is defined as: \[\begin{align} L\left( {D,{D_{area}}} \right) = \frac{1}{{{N_{area}}}}\sum\limits_{i = 1}^{N_{area}} {smoot{h_{{L_1}}}({D_{area}} - \hat{D})} \end{align}\] in which \[\begin{align} Smoot{h_{{L_1}}}(x) = \left\{ \begin{array}{l} 0.5{x^2},{\rm{ if }}\left| x \right| < 1\\ \left| x \right| - 0.5,{\rm{ otherwise}} \end{array} \right. \end{align}\] where \(N_{area}\) denotes the number of available pixels in the filtered disparity map \({D_{area}}\) and \(\hat{D}\) represents the predicted disparity. Besides, we can also employ the generated pseudo-label as supervision to train the monocular depth estimation network in an unsupervised way. The loss function is defined as: \[\begin{align} L({\hat{D}_{mono}},{D_{area}}) = \sqrt {\frac{1}{{{N_{area}}}}\sum\limits_{i = 1}^{{N_{area}}} {d_i^2 - \frac{\lambda }{{{n^2}}}} {{(\sum\limits_{i = 1}^{{N_{area}}} {{d_i}} )}^2}} \end{align}\] in which \[\begin{align} {d_i} = \log {{\hat{D}}_{mono}} - \log {D_{area}} \end{align}\] where \(N_{area}\) denotes the number of available pixels in the filtered disparity map \({D_{area}}\) and \(\hat{D}_{mono}\) represents the predicted disparity map by monocular depth estimation networks. The balancing factor \(\lambda\) is set to 0.85. Then the disparity can be converted to depth by triangulation: \[\begin{align} Depth = \frac{{fB}}{d} \end{align}\] where f denotes the camera’s focal length and B is the baseline, i.e., the distance between two camera centers.

4 Stereo matching Experiments↩︎

4.1 Dataset↩︎

SceneFlow: This is a large synthetic dataset including 35,454 training and 4,370 test images with a resolution of \(960\times 540\) for optical flow and stereo matching. We use it to pre-train our network.

Middlebury: Middlebury [5] is an indoor dataset with 28 training image pairs (13 of them are additional training images) and 15 testing image pairs with full, half, and quarter resolutions. It has the highest resolution among the three datasets and the disparity range of half-resolution image pairs is 0-400.

KITTI 2012&2015: They are both real-world datasets collected from a driving car. KITTI 2015 [7] contains 200 training and another 200 testing image pairs while KITTI 2012 [6] contains 194 training and another 195 testing image pairs. Both training image pairs provide sparse ground-truth disparity and the disparity range of them is 0-230.

ETH3D: ETH3D [8] is the only grayscale image dataset with both indoor and outdoor scenes. It contains 27 training and 20 testing image pairs with sparsely labeled ground truth. It has the smallest disparity range among the three datasets, which is just in the range of 0-64.

4.2 Implementation Details↩︎

We use PyTorch to implement our network and employ Adam (\({\beta _{\rm{1}}} = 0.9,{\beta _2} = 0.999\)) to train the whole network in an end-to-end way. The batch size is set to 16 for training on 4 Tesla V100 GPUs and the whole disparity search range is fixed to 256 during the training and testing process. \({N^1}\) and \({N^2}\) are set as 12 and 16, respectively. Threshold \(\delta\) of the ground truth uncertainty mask is 1. Asymmetric chromatic augmentation and asymmetric occlusion [34] are employed for data augmentation. Below we will introduce our training process for each term of generalization in detail.

Cross-domain generalization training: Our pre-training process on the synthetic dataset (source domain) can be broken down into three steps. Firstly, we use switch training strategy to pre-train our UCFNet in the SceneFlow dataset. Specifically, we first use ReLU to train our network from scratch for 20 epochs, then we switch the activation function to Mish and prolong the pre-training process in the SceneFlow dataset for another 15 epochs. Secondly, we fix the weights of the UCFNet (except the refinement module) and train the attention-based refinement network individually for 20 epochs with a 0.0001 learning rate. Thirdly, we fix the weights of the UCFNet and train the area-level uncertainty estimation network alone for 15 epochs. The initial learning rate is 0.001 and is down-scaled by 2 after epochs 10,12,14.

Adapt generalization training: After obtaining a strong pre-training model, we can further adapt our pre-trained model to the new domain with the generated pseudo-labels. Specifically, the training process of domain adaptation can be broken down into two steps. First, we feed the synchronized stereo images of the target domain into the pre-trained model and employ the proposed uncertainty-based pseudo-label generation method to generate corresponding pseudo-labels. Secondly, we employ the generated pseudo-labels as supervision to adapt the pre-trained model to the new domain. Specifically, we first fix the weights of the refinement network and train UCFNet on the target domain for 50 epochs with a 0.001 learning rate. Then, we fix the weights of the UCFNet (except the refinement module) and train the attention-based refinement network individually for 50 epochs with a 0.0001 learning rate.

Joint generalization training: We propose a three-stage finetuning strategy for joint generalization training. First, as mentioned in the cross-domain generalization training, we employ the switch training strategy to pre-train our model in the SceneFlow dataset. Second, we jointly finetune our pre-train model on four datasets, i.e., KITTI 2015, KITTI2012, ETH3D, and Middlebury for 400 epochs. The initial learning rate is 0.001 and is down-scaled by 10 after epoch 300. Third, we augment Middlebury and ETH3D to the same size as KITTI 2015 and finetune our model for 50 epochs with a learning rate of 0.0001. The core idea of our three-stage finetune strategy is to prevent the small datasets from being overwhelmed by large datasets. By augmenting small datasets at stage three and training our model with a small learning rate, our strategy makes a better trade-off between generalization capability and fitting capability on three datasets.

4.3 Robustness Evaluation↩︎

In this section, we evaluate our method on three terms of generalization and compare it with state-of-the-art methods in each category.

Cross-domain Generalization: As the target domain data cannot be easily obtained in many real scenarios, the network’s ability to perform well on unseen scenes is indispensable for robust stereo matching. Towards this end, we evaluate methods’ cross-domain generalization by training on synthetic images and testing on real images. As shown in Tab. 2, our method far outperforms domain-specific methods [19], [20], [24], [36] and our conference version CFNet on all four datasets with a large margin. Specifically, the error rate on KITTI 2012, KITTI 2015, Middlebury, and ETH3D has been decreased by 4.26%, 10.34%, 7.80%, and 17.24%, respectively compared to CFNet. Moreover, ITAS-CFNet [49], a specially designed stereo matching method developed from our conference version for cross-domain generalization, is the current best-published method. Our method can achieve comparable performance with it on most datasets, which further verifies our uncertainty-based cascade and fused cost volume representation is an efficient approach for robust stereo matching.

Adaptation Generalization: Comparing with collecting accurate ground-truth disparities, unlabeled target data is much easier to obtain. Thus, how to employ the knowledge of unlabeled target data adapting pre-trained models to the new domain is also essential. We evaluate such adapt generalization by training on synthetic images and finetuning on unlabeled real images. As shown in Tab. 2, although the generalization of our pre-trained model has outperformed most domain generalization methods, using the knowledge of unlabeled target data can still achieve a tremendous gain and surpass all domain generalization methods. Specifically, compared to our pre-trained model UCFNet_pretrain, UCFNet_adapt achieves 37.78%, 40.38%, 20%, 37.5% error reduction on KITTI2012, KITTI2015, Middlebury, and ETH3D, respectively. Moreover, compared to the current best-published domain adaptation method AdaStereo [29], [50], our method can still outperform it on three of four datasets, which further proves the effectiveness of the proposed method. Note that our method doesn’t employ the non-adversarial progressive color transfer and cost normalization proposed in AdaStereo, thus, the performance of our method has the potential for further improvement.

Joint Generalization: Learning-based methods are usually limited to specific domains and cannot get comparable results on other datasets. Thus, the network’s ability to perform well on a variety of datasets with the same model parameters is essential for current methods. This is also the goal of Robust Vision Challenge 2020. Towards this end, we evaluate methods’ joint generalization by their performance on three real datasets (KITTI, ETH3D, and Middlebury) without finetuning. We list the result of Robust Vision Challenge 2020 in the upper section of Tab. 1. It can be seen from this table that HSMNet_RVC [34] ranks first on the Middlebury dataset. But it can’t get comparable results on the other two datasets (3rd on ETH3D 2017 and 6th on KITTI 2015). In particular, its performance on KITTI 2015 dataset is far worse than the other five. This is because this method is specially designed for high-resolution datasets and can’t generalize well to other datasets. The similar situation also appeared on other methods (GANet_RVC, CVANet_RVC, and AANet_RVC). In contrast, our conference version CFNet_RVC shows great generalization ability and performs well on all three datasets (2nd on KITTI 2015, 1st on ETH3D 2017, and 2nd on Middlebury 2014) and achieves the best overall performance. Additionally, we further compare the proposed UCFNet_RVC with the top three methods in the previous Robust Vision Challenge in the lower part of the tab. 1. As shown, our approach outperforms Deeppruner_ROB and iResNet_ROB on all three datasets with a remarkable margin. Compared with our conference version CFNet_RVC, the proposed UCFNet_RVC can achieve similar performance on Middlebury and surpass it on the other two datasets, which further verifies the effectiveness of the proposed refinement module. See corresponding visualization results in Fig. 11.

4.4 Results on KITTI Benchmark↩︎

Although our focus is not on domain-specific performance, we still fine-tune our model on KITTI 2012 and KITTI 2015 benchmarks to show the efficiency of our method. Note that the training strategy is same with our conference paper and the only difference is the proposed attention-based disparity refinement network (please see our conference paper for more details). Specifically, some state-of-the-art real-time methods and best-performing approaches are listed in Tab. ¿tbl:tab:32KITTI32benchmark?. We find that our method achieves a 1.49% three-pixel error rate on KITTI2012, a 6% error reduction from our conference version[9] with a similar running time. Moreover, Lac-GaNet is the best published method on KITTI2012 and the proposed method can achieve comparable performance with 9 times faster speed, i.e., 1.42% (1.8s) vs 1.49% (0.21s), which implies the effectiveness and efficiency of the proposed method. A similar situation can also be observed in the KITTI2015 benchmark.

4.5 Ablation Study↩︎

To verify the effectiveness of different modules, we set a series of experiments in this section. Note that all methods are trained on the synthetic dataset first and then finetuned on the unlabeled target dataset with the proposed pseudo-labels. Generally, seven types of experiments have been executed here.

Attention-based refinement module: In the attention-based disparity refinement module, we only employ initial disparity estimation as input. Here, we test the impact of adding multi-modal input, i.e., left image, left image feature, and pixel-level uncertainty map. As shown in Tab. [tbl:tab:32abstudy95refinement], adding multi-modal input doesn’t bring noticeable gain and we can obtain the best performance by only using initial disparity estimation as input.

Domain adaptation with self-generated pseudo-label: Two terms of proxy labels, i.e., \({D_{pixel}}\) and \({D_{area}}\) can be generated by pixel-level uncertainty estimation and area-level uncertainty estimation, respectively. Here, we test the impact of each proxy label for domain adaptation individually. As shown in the Tab. [tbl:tab:32abstudy95pixel95area], the two terms of proxy labels can both promote the performance of the pre-training model on the target dataset and the improvement is consistent on both CFNet and UCFNet, which verifies the effectiveness of the proposed uncertainty-based pseudo-labels generation method. Moreover, the proposed area-level proxy label can achieve a larger gain due to the leveraging of multi-modal input and neighboring pixel information, e.g., the d1_all error rate of CFNet can further decrease from 4.5% to 3.88% after employing \({D_{area}}\) as supervision.

Threshold of proxy label generation: Threshold \(t\) is an essential hyperparameter in proxy label generation, which controls the density and reliability of the filtered proxy labels. Hence, we made a detailed ablation study to analyze the impact of different threshold settings for domain adaptation. As shown in Tab. 3, we show the tradeoff between accuracy and density of generated proxy labels in different threshold settings and the corresponding domain adaptation results. It can be seen from the table that the proposed uncertainty estimation can effectively evaluate the confidence of current disparity estimations, e.g., by removing 21.91% of uncertain pixels (\(U_{area}>0.5\)), we decrease the D1_all error rate by 72.12% (from 5.2% to 1.45% in KITTI 2015 training set). Moreover, we can find that a more accurate while sparser pseudo-label is not always good for final domain adaptation performance and we should seek a suitable balance between accuracy and density, i.e., threshold 0.9 for pixel-level uncertainty estimation and threshold 0.2 for area-level uncertainty estimation in our experiment setting.

Pixel-level uncertainty estimation vs area-level uncertainty estimation: Inspired by the standard evaluation metric in confidence estimation [56], [57], we propose to use the ROC curve and its area under curve (AUC) to quantitatively evaluate the performance of the proposed pixel-level and area-level uncertainty estimation. Specifically, we firstly sort pixels in the predicted disparity map following decreasing order of uncertainty. Then, we compute the D1_all error rate on sparse maps obtained by iterative filtering (e.g., 5% of pixels with higher uncertainty each time) from the dense map and plot the ROC curve, whose AUC quantitatively assesses the confidence effectiveness(the lower, the better). The ROC curve of the proposed method is shown in Fig. 13. Note that we also plot the roc curve of the traditional uncertainty estimation method, i.e., image-level and feature-level left-right consistency check [17], [58] to further show the effectiveness of the proposed method. As shown in Fig. 13, the proposed pixel-level and area-level uncertainty estimation can both generate a more accurate disparity map than the traditional left-right consistency check at any density and the area-level uncertainty estimation can achieve the best performance. Visualization of generated proxy labels is shown in Fig. 12. As shown, the generated two terms of uncertainty map (sub-figs (d) and (g)) are highly correlated with the error map. Hence, we can employ the proposed uncertainty estimation to filter out the high-uncertainty pixels of the original estimation and generate reliable pseudo labels. Moreover, the comparison between sub-figs (e) and (h) further shows the superiority of the proposed area-level uncertainty estimation. As shown, our area-level uncertainty estimation \({D_{area}}\) can employ the neighboring pixel information to better preserve the instance-level correct disparity estimation result, e.g., the pedestrian and pole of the input image.

Iteration number of domain adaptation: Our result can further be improved by iterative domain adaptation. Take two times of iteration as an example. At iteration 1, we employ the proxy label generated by the pre-training model to finetune the pre-training model. Then, at iteration 2, we can further finetune the pre-training model by employing the finetuned model to generate better proxy labels. More specifically, the generated pseudo-labels are employed to adapt the main network, refinement module, and area-level uncertainty estimation network one by one in each iteration. It can be seen from Tab. ¿tbl:tab:32iteration? that iterative domain adaptation can significantly improve the performance of disparity estimation on the target dataset, e.g., employing two iterations can decrease the D1_all error rate from 3.64% to 3.14%. Note that, using more iterations cannot further improve estimation accuracy due to the optimization of pseudo-label has an upper limit by iterative domain adaptation.

Area-level uncertainty estimation: In the area-level uncertainty estimation module, we employ multi-model input, i.e., pixel-level uncertainty map, initial disparity map, and left image to drive our network better evaluate the uncertainty of current disparity estimation. Here, we test the impact of each input individually. All methods are trained on the scene flow training dataset and then tested on the unseen kitti2015 dataset. As shown in Tab. [tbl:tab:32abstudy95area-level], the result verifies all the multi-modal inputs work positively to filter out matching-error points and compared with other inputs, the pixel-level uncertainty map achieves the largest gain.

Data argumentation: We also test the influence of removing additional data augmentation, e.g., asymmetric chromatic augmentation and occlusion on two real datasets. As shown in Tab. 4, the proposed method can still achieve a comparable result with the state-of-the-art method on both two terms of generalization even without data augmentation. For example, AdaStereo is the best-published domain adaptation method and indeed uses additional data augmentation. However, the proposed no data augmentation version UCFNet_adapt* can still obtain comparable results on both ETH3D and KITTI datasets. Moreover, data augmentation works positively to improve the performance on all evaluation metrics and we recommend deploying it in the final model.

Additionally, as our original cascade and fused cost volume representation (CFNet) is mainly designed for joint generalization, we also perform various ablation studies to show the effectiveness of each network design in joint generalization evaluation. We divide 5 images from each real dataset (KITTI 2015, Middlebury, and ETH3D) as a validation set and use the rest of them as a training set to finetune our pretrain model. Results are shown in Tab. 5. Below we describe each component in more detail.

Feature extraction: We compare our pyramid feature extraction with the most widely used Resnet-like-network [19], [24]. As shown, our pyramid feature extraction can achieve similar performance with a faster speed, likely because the employing of small scale features is also helpful in feature extraction.

Cost volume fusion: We fuse three small-resolution cost volumes to generate the initial disparity map. Here, we test the impact when only a single volume is used. Cost volume fusion can achieve better performance with a slight additional computational cost.

Cost volume cascade: We test three ways of generating the next stage’s disparity searching space in cascade cost volume representation. As shown, learned parameters based pixel-level uncertainty estimation achieves the best performance with tiny additional computation complexity.

Finetuning strategy: We test three terms of finetuning strategy. As shown, neither directly augmenting small datasets at the beginning (two-stages) nor only extending the number of iterations (three stages_no augment) can improve the accuracy of predictions on small datasets. Instead, our strategy can greatly alleviate the problem of small datasets being overwhelmed by large ones.

4.6 Extreme Situation↩︎

As mentioned in Sec. 4.3, our pre-training model has strong cross-domain generalization and can generate relatively reasonable results for the subsequent pseudo-label generation. However, it still cannot work in some extreme situations, e.g., pictures taken inside a tunnel. As shown in Fig. 14, we give some extreme cases, in which our pre-training method predicts a totally wrong result. As for such cases, the proposed pixel-level uncertainty estimation (sub-figs (f)) method is difficult to filter out all errors in the disparity estimation while area-level uncertainty estimation (sub-figs (i)) can achieve this goal and generate a disparity map without valid pixels, making the noisy labels don’t affect subsequent domain adaptation. Moreover, the proposed UCF_adapt can indeed predict a reasonable result in such extreme situations (sub-figs (m)) even if we don’t have corresponding valid proxy labels for these extreme cases, which further verifies the effectiveness of the proposed domain adaptation method.

5 Monocular depth estimation Experiments↩︎

Recall our goal is to push methods to be robust and perform well across different datasets without using the ground truth of the target domain. This is same in the monocular depth estimation setting. Indeed, as the monocular depth estimation is an ill-posed problem, it even needs more annotated data. However, acquiring labeled real-world data is cumbersome and costly in most practical settings, e.g., expensive LiDAR with careful calibration is required to obtain depth ground truth in outdoor scenes. Instead, stereo matching is a cheaper option. Hence, we propose to explore the feasibility of eliminating the need for lidar and using stereo matching methods to collect ground truth data. That is the proposed stereo matching method is served as the offline ground truth collection system and the monocular depth estimation network is the deployed online depth estimation module. Following this motivation, we select to use the stereo matching model trained on the synthetic dataset and unlabeled target domain data (UCFNet_adapt) to generate the pseudo-label for the training of monocular depth estimation. Note that both state-of-the-art supervised and self-supervised monocular depth estimation approaches are compared here.

5.1 Dataset↩︎

We use the KITTI dataset [6] as the training dataset which consists of calibrated videos registered to LiDAR measurements of city scenarios. The depth evaluation is done on the LiDAR pointcloud. Following [[59]][60][61], seven standard metrics, named “Abs Rel”, “Sq Rel”, “RMSE”, “RMSE log”, “\(\delta < 1.25\)”, “\(\delta < 1.25^2\)” and “\(\delta < 1.25^3\)” are used to evaluate the performance of the predicted depth information. Tab. 6 demonstrates the definition of each evaluation metric, and please see [59] for evaluation details. Training mode: According to different training modes, three kinds of results are provided here. (1). Results of self-supervised monocular depth estimation approaches, which do not need the supervision of ground truth; (2). Results of supervised monocular depth estimation approaches with ground truth as supervision; (3). Results of supervised monocular depth estimation approaches with generated pseudo-labels as supervision. Moreover, as these methods eliminate the need for ground truth, they can be regarded as unsupervised approaches.

5.2 Implementation Details↩︎

We select two representative supervised monocular depth estimation models LapDepth\(\footnote{\href{https://github.com/tjqansthd/LapDepth-release/} {LapDepth} uses GPL-3.0 License and we download the code from their official GitHub website.}\) and BTS\(\footnote{\href{ https://github.com/cleinc/bts/}{BTS} uses GPL-3.0 License and we download the code from their official GitHub website.}\) to verify the effectiveness of the proposed pseudo label in monocular depth estimation. All settings remain the same as in their original paper except that we employ the generated pseudo-labels rather than ground truth to supervise the network. Specifically, the whole training process can be divided into three steps: Firstly, we employ the synthetic data (source domain) and the unlabeled stereo images of the real dataset (target domain) to train a robust stereo matching network UCFNet_adapt. Secondly, we feed the synchronized stereo images of the KITTI raw dataset into the UCFNet_adapt and employ the proposed uncertainty-based pseudo-label generation method to generate corresponding pseudo-labels. Thirdly, we employ the generated pseudo-labels as supervision to train the selected two representative supervised monocular depth estimation methods: LapDepth and BTS.

5.3 Comparisons among different training modes↩︎

Tab. 7 demonstrates the results among different training modes. In specific, pink areas mean results obtained by self-supervised approaches, purple areas mean results obtained by supervised approaches with ground truth supervision, and green areas mean results obtained by supervised approaches with our generated pseudo-labels. “*” means results evaluated by the official annotated ground truth of KITTI, where the ground truth is obtained by combing multi-frame of the point cloud, while default (without “*”) means results evaluated by raw point cloud data of KITTI.

Note that, the ground truth is not needed in the generation of pseudo-labels, therefore, approaches supervised with our generated pseudo-labels (Bts_pseudo_label, Bts_pseudo_label_full, lapdepth_pseudo_label and lapdepth_pseudo_label_full in Tab. 7) can be regarded as unsupervised approaches.

The comparison between state-of-the-art self-supervised and supervised monocular depth estimation methods is shown in Tab. 7. It can be seen from the table that the proposed pseudo-label-based method (green areas) outperforms self-supervised monocular depth estimation approaches (pink areas) by a large margin, which proves that our generated pseudo-labels can well supervise monocular depth estimation approaches. Moreover, due to the usage of ground truth, the performances of supervised depth estimation approaches (purple areas), such as DORN [65], Bts [66], and lapdepth [67], commonly outperform self-supervised based approaches (pink areas). In this paper, we claim that pseudo-label-based methods can achieve comparable or even better results than supervised approaches without the need for ground truth. Specifically, as shown in the green areas of Tab. 7, Bts [66] and lapdepth [67] denote the original result of two representative supervised monocular depth estimation models trained by ourselves with the official implementation. Bts_pseudo_label and lapdepth_pseudo_label denote we employ the pseudo-labels of the training dataset to supervise the two representative supervised monocular depth estimation models. Note that the only difference between the proposed pseudo-label-based methods and the corresponding original implementation is the supervision signals, i.e., pseudo-labels vs ground truth. As shown, Bts_pseudo_label and lapdepth_pseudo_label can achieve comparable results with the supervised version, e.g., the rmse of Bts_pseudo_label in default evaluation is 3.730, which is only 0.97% higher than BTS. Similar situations can also be observed in lapdepth_pseudo_label. Note that we also observe that some previous work, such as SD-SSMDE [68] and monoResMatch [32] also explore using pseudo-label generated by traditional stereo matching methods [32] or self-distillation [68] to supervise the monocular depth estimation method. However, these methods still have a large gap between supervised depth estimation approaches and cannot achieve comparable results with the proposed uncertainty-based pseudo-label. Additionally, as the generation of pseudo-labels is not dependent on ground truth, pseudo-labels of the testing dataset can also be generated, which can be combined into the training dataset for better performance. Bts_pseudo_label_full and lapdepth_pseudo_label_full in Tab. 7 demonstrate the corresponding results. We can see that Bts_pseudo_label_full and lapdepth_pseudo_label_full can greatly improve the performances of Bts_pseudo_label and lapdepth_pseudo_label on all evaluation metrics. Moreover, they can even outperform the fully supervised method Bts [66] and lapdepth [67] with large margins, which further verifies our hypothesis that stereo matching can be a viable alternative to reduce the cost of ground truth collection in the monocular depth estimation setting.

Qualitative comparison results on the KITTI Eigen test split are shown in Fig. 15. Specifically, scenes with different depths of field are provided in which red denotes a smaller depth. As shown, our method can better distinguish objects in both foreground and background areas (see dash boxes in the pictures). Moreover, supervised methods generally cannot generate reasonable results on unlabeled areas, e.g., the sky region and the upper part of the scenes. This is mainly caused by the limitation of LIDAR, e.g., the ground truth obtained by lidar is very sparse and cannot collect valid data in the upper region of the scene. Instead, the proposed method can provide denser pseudo-labels and cover all regions in the image, thus significantly improving the visualization results on unlabeled areas. (see green dash boxes in the picture).

6 Conclusion↩︎

We have proposed an Uncertainty based cascade and fused cost volume representation for robust stereo matching. Specifically, a fused cost volume is proposed to alleviate the domain shifts and a cascade cost volume is employed to balance different disparity distributions, where pixel-level uncertainty estimation is at the core. We use it to adaptively narrow down the next stage’s pixel-level disparity searching space. Then, we propose an uncertainty-based pseudo-labels generation method to further narrow down the domain gap. By the cooperation between pixel-level and area-level uncertainty estimation, we can obtain a sparse while reliable pseudo label for domain adaptation without the need for ground truth. Experiment results show that our proposed method can achieve strong cross-domain, adapt, and joint generalization and obtain the 1st place on the stereo task of Robust Vision Challenge 2020. Moreover, our uncertainty-based pseudo-labels can be extended to train monocular depth estimation networks in an unsupervised way and even achieves comparable performance with the supervised methods.

Acknowledgments↩︎

This research was supported in part by the National Natural Science Foundation of China (62271410), Jiangxi Natural Science Foundations (20224BAB212009), and the Fundamental Research Funds for the Central Universities.

Acknowledgment↩︎

References↩︎

---

1. http://www.robustvision.net/rvc2020.php↩︎