1 Introduction↩︎

While the human visual system is capable of intuitively performing robust scene understanding and perception. The reasoning of occlusion relation is highly challenging for machines. Depending on the number, category, orientation, and position of objects, the boundaries of objects are elusive; no simple priors can be applied to recover the foreground and background in the scene. Occlusion relationship reasoning of objects from monocular images reveals relative depth differences among the objects in the scene, which is fundamental in computer vision applications, such as mobile robots [1]–[3], object detection [4]–[8], segmentation [4], [9]–[14], monocular depth estimation/ordering [15]–[20], and 3D reconstruction [21]–[23].

Traditional methods [18], [24]–[27] extract the occlusion boundary and infer occlusion relation by exploiting low-level visual cues with hand-crafted features, which is not effective due to the difficulty of defining boundaries and occlusion cues. Modern convolution neural networks (CNNs) [28]–[30] have significantly boosted the performance of occlusion relationship reasoning by commonly decomposing the task into two subtasks: occlusion boundary extraction and occlusion orientation prediction. The former aims to extract object boundaries from the image, while the latter targets are discovering the orientation relation. The occlusion relationship is then recovered by progressively accumulating the orientation information on the extracted boundaries. Despite the leap in occlusion relationship reasoning, we observe that two critical issues have rarely been discussed, which greatly impedes performances of previous work.

The first issue is the lack of exploitation of the limited amount of coupling between the two subtasks. On one hand, shared global properties of the image and abstract high-level semantic features act as an effective initialization for the networks to learn low-level visual cues. Shared high-level features help different branches to generate consistent outputs under the same semantic guidance while different features may produce semantically misaligned results in the fusion stage. On the other hand, the two subtasks need to learn diverse and concrete properties in the lower stage with larger spatial size. Specifically, occlusion boundary extraction emphasizes more on locating boundaries, while occlusion orientation attaches more importance to exploiting relations between areas. In a word, the two subtasks are inherently coupled in abstract global features and decoupled in task-specific low-level features. However, previous architectures adopt a fully shared decoder [29], [30], resulting in an initialization feature map from the lower network stage.

The second issue is the lack of a good representation of occlusion orientation. DOC [28] first proposed to use pixel-wise continuous orientation variable as a representation, which however makes the design of loss function complex due to angle periodicity. DOOBNet [29] then proposed to truncate the variable into range \((-\pi, \pi]\), which caused serious endpoint errors: angles close to \(-\pi\) and \(\pi\) produce large loss value while remaining relatively close orientation. This leaves the training process problematic. Therefore, designing a proper occlusion orientation representation remains challenging in the occlusion relationship reasoning.

To tackle the first issue, we propose an Occlusion-shared and Path-separated Network (OPNet), which uses abstract feature maps with smaller spatial sizes for weight sharing and is then split into two separate task-specific paths, namely the occlusion boundary extraction path and occlusion orientation prediction path. Besides, multi-scale supervision [31] is adopted in the boundary extraction path to enhance the multi-scale features. To solve the second representation dilemma, we propose a simple and robust orthogonal occlusion representation (OOR), which represents orientation with two orthogonal vectors in horizontal and vertical directions, respectively. Experiments show our method is effective and boosts the performances of both object boundary extraction and occlusion orientation prediction.

In a nutshell, our contributions are three-fold:

• We rethink the inherent properties of occlusion relationship reasoning, associated with two sub-tasks: occlusion boundary extraction and occlusion orientation regression. This motivates our advocate of a novel Occlusion-shared and Path-separated Network (OPNet) embodied with discrimination capability and expressiveness for visual occlusion reasoning as an alternative.

• We further propose the robust orthogonal occlusion representation(OOR) for predicting the occlusion orientation, which resolves the endpoint error and angle periodicity dilemma.

• The method works well on both PIOD [28] and BSDS ownership datasets [32], offering significantly better performances than the counterparts.

2 Related Work↩︎

Deep learning based occlusion relationship reasoning has achieved great success in recent years [28]–[30]. DOC [28] represents occlusion relation by using a binary boundary indicator. It exploits local and non-local image cues to learn this representation and hence recovers the occlusion relation. DOOBNet [29] proposes to address extreme boundary/non-boundary class imbalance by up-weighting the loss contribution of false negative and false positive examples with a novel Attention Loss function. OFNet [30] builds on top of the encoder-decoder structure and side-output utilization. A fusion module is designed to precisely locate the object regions from the occlusion cues.

However, these methods either use two separate architectures for boundary extraction and orientation prediction or over-share local spatial information in shallow stages. Instead, our OPNet only shares features in deep stages of the decoder and adopts a multi-scale structure, as shown in Figure 1. We also propose an orthogonal orientation representation to tackle endpoint error and angle periodicity dilemma.

3 The OPNet↩︎

In this section, we detail our network architecture, Occlusion-shared and Path-separated Network (OPNet), which is composed of a shared encoder and our proposed novel Occlusion-shared and Path-separated decoder (OPDecoder). Following previous methods, we choose a ResNet50 [33], [34] pretrained on ImageNet as our encoder. Our OPDecoder is equipped with Occlusion-shared Module (OSM), Orthogonal Perception Module (OPM), and two separated paths, i.e, a boundary path and an orientation path. Extracted boundary map from boundary path and predicted orientation map from orientation path are then merged by Non-Maximum Suppression(NMS) to produce the final occlusion relationship map in the evaluation stage.

3.1 The Occlusion-shared and Path-separated Decoder↩︎

We define the deep stages in an encoder/decoder architecture as the ones where the feature map is deeper but spatially smaller, such as res4 or res5 layer in a ResNet architecture. Such deep stages contain less parameters and spatial details, thus are rich in abstract and global information that can be well adapted to downstream tasks. In contrast, shallow stages produce spatially-large feature maps with reduced channels. As a result, they are beneficial for learning task-specific low-level details. Our OPDecoder fully utilizes such distinct features at multiple stages.

3.1.0.1 Shared Deep Stage and Separated Shallow Stages

Both boundary extraction and occlusion orientation prediction are dense prediction tasks aiming to recover pixel-wise spatial details with the understanding of high-level occlusion information. The former focuses on locating, while the latter expresses the relationship between regions where occlusion occurs. We thus propose our OPDecoder with a deep-to-shallow structure. Specifically, in order to better consider the connection and distinction between the two tasks, we divide the decoder into two parts, the shared deep stage with joint occlusion information and the separate shallow stages with concrete spatial details. The shared high-level semantic features in the deep stage learn more abstract occlusion cues by accepting supervision signals from both the boundary path and the orientation path. The boundary extraction path pays more attention to the spatial details, while the orientation regression path focuses on recovering regional relationships. Excessive sharing could result in mutual interference between the two tasks. Therefore, we also adopt separate structures in the shallow stages.

As shown in Figure 2 (a), our decoder contains a deep occlusion-shared phase and a shallow spatial information branching phase. In the occlusion-shared phase, we use OSM to decode and aggregate two deepest stages’ features and then pass them to the subsequent boundary path and orientation path. In the shallow branching phase, considering the possible interference between the two tasks, we use OPM to enhance the features from shallow layers of the encoder, which will be fed to the following two paths.

In the boundary extraction path, the shared feature map from OSM is progressively aggregated with transformed encoder features from deep to shallow stages. To better utilize spatial information at different scales, multi-scale boundary maps are generated, which are then fused together to extract a merged boundary map. Multi-scale supervision is applied to each generated boundary map.

Different from the boundary path, the orientation path is deliberately designed to only accept skip-connected features from the res2 layer to exploit less shared spatial details in shallow layers. It only keeps one supervision signal in the original scale due to the ambiguity of scaled orientation maps.

Orthogonal Perception Module (OPM) As shown in Figure 2 (b), OPM contains three branches. The center branch consists of two \(3 \times 3\) convolution blocks, while the other two are composed of two \(1 \times 3\) and \(3 \times 1\) strip convolution blocks, respectively. Figure 3 shows the receptive fields of module output features. The two stripe convolution paths simultaneously increase the receptive field and feature weight of the center pixel in orthogonal directions, and hence accumulate more informative orthogonal signals for further prediction of the OOR.

Occlusion Shared Module (OSM) As shown in Figure 2 (c), OSM aggregates shared high-level features by fusing the upsampled res5 feature map and res4 feature map. It is designed to only merge features from deep stages, retaining abstract high-level shared occlusion features. OSM accepts supervision signals from both the boundary path and the orientation path to act on the high-level semantic features with larger receptive fields.

3.2 Orthogonal Occlusion Representation↩︎

In this section, we first point out the disadvantages of existing occlusion representations, i.e. angle periodicity and endpoint error. Different representation methods are then visualized for intuitive illustration. Finally, we present the novel orthogonal occlusion representation, which avoids the aforementioned two disadvantages by using two orthogonal vectors. Figure 4 shows a schematic diagram of four boundary-based representations of occlusion orientation, the lower left and upper right corners are the foreground and the background, respectively, while the diagonal line is the occlusion boundary between the two regions.

Figure 4 (a) shows the left rule of indicating foreground and background in an image. The occlusion boundary is represented with an arrow, the left-hand side of which is the foreground. Figure 4 (b) is the label classification method based on complete boundaries commonly adopted by the traditional methods [24], [35]–[38], which classifies the obtained occlusion boundaries into two types. Label 1 indicates that the orientation of the boundary is from the starting point to the ending point, while label 2 means the opposite. This method is considered ineffective due to the fact that detected boundaries are sparse and sometimes discontinuous. Figure 4 (c) illustrates pixel-level orientation variable representation proposed in DOC [28], which predicts boundary orientation by predicting a continuous orientation variable \(\theta \in (-\pi, \pi]\) for each pixel in the image. This method can well adapt to convolution blocks with dense prediction attributes and avoid the dependency on the quality of boundaries. It brings significant performance improvement on occlusion relationship reasoning and is widely adopted by CNN-based methods [28]–[30], [39].

However, pixel-level orientation variable representation still has two critical issues. Firstly, it requires the network to regress an accurate angle for each boundary pixel, whereas predicting accurate angle variables brings an unnecessary burden. As long as the predicted angle has an error less than \(\pi / 2\), the foreground-background relationship can be correctly recovered. The second issue is the negative impact of angle periodicity on loss functions. The period of the angle variable is \(2\pi\), meaning that two angles whose difference is a multiple of \(2\pi\) are equivalent. However, such equivalence is hard to maintain in the loss function. In order to solve this problem, DOC [28] defines a periodic segmentation function to represent the difference between two angles. But experiments show that this discontinuous loss function hinders the learning of the network. DOOBNet adopts the strategy of truncating the predicted value to \((-\pi, \pi]\), which however brings serious endpoint errors, i.e. angles near \(-\pi\) and \(\pi\) could result in the higher loss while in fact staying close, introducing unnatural supervision signals near the endpoints.

We hereby propose our orthogonal occlusion representation (OOR): using a pair of orthogonal vectors, \(\vec{a}\) along the horizontal axis and \(\vec{b}\) along the vertical axis, to represent occlusion orientation. As shown in Figure 4 (d), both \(\vec{a}\) and \(\vec{b}\) point to the background near the occlusion boundary. OOR simplifies the prediction of occlusion orientation by instead pointing out the background both horizontally and vertically, greatly enhancing the robustness of prediction. Compared with previous the orientation variable representation, our OOR is simpler and works better, bypassing both angle periodicity and endpoint error dilemma.

Figure 5 visualizes the feature maps extracted from the Res2 layer trained with different orientation representations. Compared with the feature maps obtained by using DOOBNet’s orientation representation and loss, the contrast between foreground and background of our result is more obvious. Experiments in Table. ¿tbl:res95table? show that our method improves all three occlusion-related metrics (O-ODS, O-ODS, O-AP) by at least \(2\%\) upon orientation representation of DOOBNet, which shows stronger occlusion perception potential of our OOR. Figure 6 shows the comparisons of different methods on occlusion orientation recall towards different boundary directions. We divide \((-\pi, \pi]\) into eight bins and calculate the recall of orientation prediction in each bin. Results show that DOC’s and DOOBNet’s orientation representations and loss pay too much attention to the vertical direction and produce large errors near two endpoints due to the aforementioned endpoint error dilemma.

3.3 Network Training↩︎

For an input image \(I\), the ground truth label is represented as a pair of object boundary map and occlusion orientation map \(\{Y, O\}\), where \(Y = (y_j, j = 1, \dots ,|I|), y_j \in \{0, 1\}\). \(O = (\theta_j, j = 1, \dots ,|I|), \theta_j \in (-\pi, \pi]\) and \(\theta_j\) is invalid when \(y_j=0\). In CNN-based methods of occlusion relationship reasoning, occlusion boundary extraction and occlusion orientation prediction are trained simultaneously. Thus the overall loss \(\mathcal{L}\) is formulated as: \[\begin{align} \mathcal{L} = \mathcal{L}_{\text{B}}\end{align} + w_{\text{O}} \cdot \mathcal{L}_{\text{O}},\] where \(w_{\text{O}}\) is the weight for occlusion orientation loss. \(\mathcal{L}_{\text{B}}\) and \(\mathcal{L}_{\text{O}}\) represent the loss of boundary path and orientation path, respectively. In boundary path, we can get a collection of five side boundary maps and a fused boundary map \(\{\hat{Y}_{\text{s-1}}, \hat{Y}_{\text{s-2}}, \cdots, \hat{Y}_{\text{s-K}}, \hat{Y}_{\text{f}}\}\). We formulate the loss of boundary extraction as, where in this work \(K = 5\) : \[\mathcal{L}_{\text{B}} = w_{\text{f}} \cdot \Gamma(\hat{Y}_{\text{f}}, Y) + \sum_{i=1}^{K} w_{\text{s-i}} \cdot \Gamma(\hat{Y}_{\text{s-i}}, Y),\] where \(w_{\text{s-i}}\) and \(w_{\text{f}}\) are weights for the side boundary map loss and fused boundary map loss, respectively. \(\hat{Y} = (y_j, j = 1, \dots ,|I|), y_j \in (0, 1)\) represents the predicted boundary map . The function \(\Gamma (\hat{Y}, Y)\) is computed with all pixels. Since the distribution of boundary/non-boundary pixels is heavily biased, we employ a class-balanced cross-entropy loss. we define \(Y_{-}\) and \(Y_{+}\) as the non-boundary and boundary ground truth label sets in a mini-batch of images. We then define \(\Gamma (\hat{Y}, Y)\) as: \[\begin{align} \Gamma(\hat{Y}, Y) & = - \lambda \alpha \sum_{j \in Y_{-}} \log \left(1-\hat{y}_{j}\right)- (1-\alpha) \sum_{j \in Y_{+}} \log \left(\hat{y}_{j}\right). \end{align}\] Here, \(\alpha=\left|Y_{+}\right| /\left(\left|Y_{+}\right|+\left|Y_{-}\right|\right)\) can balance the boundary/non-boundary pixels, and \(\lambda\) controls the weight of positive over negative samples.

The function \(\mathcal{L}_{\text{O}}\) calculates the loss of the orientation only at the groundtruth boundary pixels. Based on our representation, we proposed a novel loss called Orthogonal Orientation Regression loss (OOR), which is defined as: \[\mathcal{L}_{\text{O}} = \text{OOR}(\hat{a}, \hat{b};\theta) = \operatorname{smooth}_{L_{1}}\Big(\Phi(\hat{a}, \hat{b};\theta)\Big),\] in which we design the novel formulation as: \[\begin{align} \Phi(\hat{a}, \hat{b};\theta)=(\frac{\hat{a}}{\sqrt{\hat{a}^{2}+\hat{b}^2}}-\cos \theta_{j})^{2} +(\frac{\hat{b}}{\sqrt{\hat{a}^{2}+\hat{b}^{2}}}-\sin \theta_{j})^{2}. \end{align}\] \({\hat{a}}\) and \(\hat{b}\) represent prediction value of the network. We define the \(\vec{a} = (\hat{a}, 0)\) and \(\vec{b} = (0, \hat{b})\). The \(\theta\) is the ground truth of occlusion orientation.

4 Experiment↩︎

4.1 Datasets and Implementation Details↩︎

We evaluate the proposed approach on two public datasets: PIOD [28] and BSDS ownership dataset [32].

PIOD dataset contains 9,175 images for training and 925 images for testing. Each image is annotated with groundtruth object instance boundary map and corresponding orientation map.

BSDS ownership dataset includes 100 training images and 100 testing images of natural scenes. Each image is annotated with groundtruth object instance boundary map and corresponding orientation map.

Implementation details: Our network is implemented in PyTorch. We use a ResNet50 [33], [34] architecture pretrained on ImageNet as the backbone encoder. AdamW optimizer is adopted to train our network. The learning rate is 3e-6. Training images are randomly cropped into size \(320 \times 320\) and form mini-batches of size 8. We train our network on the PIOD dataset for 40k iterations and set hyperparameter \(w_{\text{s-[2-5]}}\), \(w_{\text{s-1}}\), \(w_{\text{O}}\), \(w_{\text{f}}\) and \(\lambda\) to 0.5, 1.1, 1.1, 2.1, 1.7, respectively. we train our network on the BSDS ownership dataset for 20k iterations and set \(\lambda\) as 1.1. Other hyper-parameters are kept the same as in PIOD.

Evaluation metrics: We compute the precision and recall (PR) for boundary extraction (BPR) and occlusion orientation (OPR). Three standard evaluation metrics are then calculated from PR: fixed contour threshold (ODS), best threshold of image (OIS) and average precision (AP). In the following sections, we use B-Metric to represent metrics calculated from BPR and O-Metric to represent metrics calculated from OPR. Note that, OPR is only calculated at the correctly detected boundary pixels.

4.2 Comparison with the state-of-the-art methods↩︎

We compare our approach with recent methods including SRF-OCC [25], DOC-HED [28], DOC-DMLFOV [28], DOOBNet [29], OFNet [30].

Performance on PIOD dataset: The Precision-Recall curves of both boundary extraction and occlusion orientation tasks on PIOD dataset are shown in Figure 7 (a) and (b). Clear improvements upon OFNet can be observed on both tasks. Table [tbl:result95table] shows that even though trained alone on each subtask, our method still outperforms others by a large margin(\(3.5\%\) B-ODS and \(4.3\%\) O-ODS).

Performance on BSDS ownership dataset: The BSDS ownership dataset is difficult to train due to the small number of training samples. Nonetheless, our method can still achieve superior performance to others in both BPR and OPR as shown in Figure 7 (c)(d). As shown in Table ¿tbl:res95table?, benefited from our proposed OOR, our method achieves a huge 10% O-AP gain and an 8.3% B-AP gain compared to state-of-the-art, showing strong potential in both orientation prediction and boundary extraction.

By substituting OOR with dor/dbr (OPNet+dor/dbr in Table [tbl:result95table]), OPNet with joint training strategy outperforms previous works but underperforms even the single boundary branch training of OPNet (Ours \(\dagger\)), which verifies the superiority of our network structure and OOR, respectively.

Finally, we visualize the boundary maps and occlusion relationship maps, as shown in Figure 8. It can be observed that our method extracts more complete and clearer boundaries on multiple images than DOOBNet[29] and OFNet[30]. In complex and indistinguishable scenes, such as images on the right-hand side, we are still ahead of others, showing stronger generalization potential. As shown in the last image sets on the left-hand side, our model can even discern the leg which is not labeled on the groundtruth map. In summary, our method can outperform others by a large margin both in boundary extraction and occlusion orientation prediction, which validates its effectiveness.

4.3 Ablation Study↩︎

In this section, we conduct experiments on the PIOD dataset to study the impact of different architectural designs and verify each component in our network.

Comparison on number of shared layers: We compared sharing top-1, top-2, top-3, and top-4 stages at high stages. As shown in Table ¿tbl:shard95layer95table?, the best result appears when sharing the top-2 stages, surpassing the second best(top-1) by 2.3% in B-AP. When the sharing extends to the lower stages, the result begins to deteriorate, which empirically proves our argument that the two branches should share occlusion information existing in higher stages and remain separated in lower stages to recover task-specific spatial information.

Comparison of orientation losses: We change the output of our occlusion orientation path so that it can regress the continuous orientation value, as in previous work. We then compare the performance of predicting continuous orientation value(DOOBNet) and predicting orthogonal vectors(OOR) with our network structure. As shown in Table ¿tbl:res95table?, the proposed OOR and OOR loss surpasses DOOBNet’s orientation representation by 6.5% and 10% O-AP on PIOD and BSDS ownership dataset, respectively, which validates the effectiveness of our orthogonal orientation representation and loss.

Multi-scale supervision effect: Our boundary path outputs five side boundary maps with multi-scale supervision signals. To validate the effectiveness of such multi-scale supervision, we compared results of supervising the lowest one to the lowest five side boundary maps, where a lower boundary map has a larger spatial size. As shown in Table ¿tbl:shard95layer95table?, the performance of only supervising the lowest side is significantly worse than other settings. Performance is also greatly improved by using five-scale supervision, surpassing the second best(using 4 side maps) by 2.0% B-AP, which demonstrates the importance of multi-scale supervision and multi-result fusion.

5 Conclusion↩︎

In this paper, we present the novel OPNet, which shares occlusion features in the deep stage and splits into two separate decoder paths with larger spatial sizes. Besides, we apply multi-scale supervision to the boundary extraction. We also present a robust orthogonal occlusion representation, which avoids the endpoint prediction errors and negative impact of angle periodicity. Our network achieves new state-of-the-art performances on both PIOD and BSDS ownership datasets.

Acknowledgement↩︎

This work was supported by the national key R & D program intergovernmental international science and technology innovation cooperation project (2021YFE0101600).

References↩︎