zhangsy@travelsky.com.cn

fanglue.zhang@vuw.ac.nz

1 Introduction↩︎

Estimating surface normal is a crucial task in computer vision as it provides essential geometric information about the structure and orientation of surfaces within a scene. By offering valuable insights into the underlying 3D geometry of objects, accurate surface normal estimation contributes significantly to advancing the capabilities of a wide range of applications, including object recognition, autonomous driving, and robotics. Despite significant progress in normal estimation for conventional perspective images [1]–[3], the task of 360° surface normal estimation remains less explored. When directly applying the methods proposed for perspective images to 360° images, the results are unsatisfactory (see Fig. 5). This is because the spatial warping inherent in 360° images is not taken into account in these techniques and so we instead, in this paper, develop specialized techniques tailored to their spherical representation.

To address the distortions inherent in panoramic imagery, prior work in related tasks such as 360° depth estimation has proposed projection–fusion architectures (e.g., UniFuse [4]) that fuse complementary information from equirectangular projection images and cube-map patches to produce more accurate predictions in the spherical domain. While effective for depth, these projection-fusion strategies must reconcile a domain gap between different projections and introduce significant cross-projection fusion overhead. Alternative distortion-aware approaches embed spherical geometry directly into CNNs or adopt specialized loss formulations to compensate for spatial warping [3], [5]. More recently, self-attention mechanisms have been applied to recover holistic scene structure in the 360° domain [6], [7], with transformers demonstrating clear advantages in modelling long-range dependencies. However, despite these advances in depth estimation and global modelling, such depth-oriented and transformer-based solutions exhibit two complementary shortcomings when transferred to surface normal estimation: (1) projection-fusion and distortion-aware CNN solutions tend to prioritise correcting large-scale geometric distortions and therefore do not always preserve the high-frequency, local geometric cues required for accurate surface orientation; and (2) conventional transformer pipelines, although powerful at global reasoning, commonly lack inductive mechanisms to retain fine-grained local detail and do not explicitly exploit the spherical distribution of features. These observations — corroborated by our empirical comparisons — motivate a hybrid design that integrates CNN-based local feature extraction with multi-level, spherical-aware self-attention to jointly recover fine-scale geometry and global context for 360° surface normal estimation.

In this paper, we present PanoNormal, a framework for monocular surface normal estimation in 360° indoor environments. PanoNormal strategically integrates the strengths of CNNs for extracting low-level features and enhancing locality, along with the advantages of transformers in capturing and associating long-range dependencies. We introduce a deep learning module aimed at extracting low-level features from raw 360° images, departing from the conventional approach of direct tokenization inputs. To enhance the capabilities of self-attention in exploring the structure of the surface normal map, we propose a multi-scale transformer decoder, enriching the final representation by integrating information across diverse scales. As demonstrated in Fig. 1, previous state-of-the-art methods yield sub-optimal results when applied to complex scenes, failing to accurately identify complete object boundaries and providing incorrect surface normal vector directions, particularly in the areas highlighted by red boxes. In contrast, our approach predicts sharper and more accurate details of local areas and finely-delineated object boundaries, significantly improving the holistic geometry understanding of the scene. We conduct extensive experiments on public datasets to evaluate our approach, and the results demonstrate that PanoNormal consistently outperforms state-of-the-art models across all examined datasets. Our contributions are summarized as follows:

• We present PanoNormal, a specialized vision transformer architecture designed for estimating surface normals in monocular indoor ERP imagery. Notably, PanoNormal stands as the first panoramic transformer tailored for the 360° surface normal estimation task.

• PanoNormal seamlessly integrates the benefits of CNNs for robust low-level feature extraction, fortifying locality. Additionally, it harnesses the strengths of transformers in capturing long-range dependencies. The incorporation of a multi-scale scheme further enhances the holistic representation of the geometric structure within the scene.

• Extensive experiments conducted on widely recognized benchmarks (namely 3D60, Stanford2D3D, Matterport3D, SunCG, and Structured3D) illustrate that PanoNormal consistently surpasses state-of-the-art approaches, and contribute to further studies in this domain by providing the first comprehensive evaluation across these public benchmarks.

2 Related Work↩︎

2.1 360° Surface Normal Estimation↩︎

Most existing methods [1], [2], [8], [9] for surface normal estimation are designed for perspective images, which poses challenges, such as distortions, deformations, and domain shifts, that make conventional methods unsuitable for 360° images. Moreover, panoramic images introduce additional complexities to surface normal estimation, including variable illumination and intricate indoor layouts. Recent works can be broadly categorized into direct and indirect methods. Direct methods [3], [10], [11] introduce specific loss functions (e.g., spherical loss, hyper-sphere loss, double-quaternion loss) to comprehend the geometric structure on the unit sphere with various CNN architectures. Coors et al. [5] explicitly embed invariance against spherical distortions into CNNs, adjusting the sampling locations of convolutional filters to counter distortions and align the filters around the sphere effectively. Indirect methods [12], [13] draw inspiration from other vision tasks, e.g., 360° depth estimation, and adopt their architecture for surface normal estimation. Other work [4], [6], [7], [14] proposes methodologies that project equirectangular images into corresponding cube map or tangent plane projection sub-images for predictions or directly learn internal correlations among these projections, aiming to capture details and holistic information concurrently with distortion awareness. However, employing simple CNNs with various spherical loss functions, adopting surface normal estimation methods from depth alone, or calculating surface normals directly from depth maps [1] fails to yield satisfactory results. Besides, other studies have explored complementary directions in 360° vision. Huang et al. [15] and Kou et al. [16], [17] introduced image editing operations specifically designed for 360° representations. Wang et al. [18] investigated target scanpaths to enhance 360° media experiences, while Peng et al. [19] utilized dynamic gnomonic projection for object tracking in the 360° domain. Although these works broaden the scope of 360° research, they remain distinct from surface normal estimation. In addition, to the best of our knowledge, there are no surface normal estimation methods that provide a standardized evaluation across diverse public benchmarks, as has been established in depth estimation tasks.

2.2 Vision Transformer↩︎

Vision transformers are a class of models that use self-attention mechanisms to process images as sequences of patches [20]. ViTs [13], [21]–[23] have achieved remarkable results on conventional images captured by cameras that have a limited field of view and that have, at most, mild distortions. In contrast, 360° images offer a complete, immersive view of the surrounding scene but require adaptations and modifications of the original ViTs to handle challenges such as significant image distortions, object deformation, and domain shifts. Shen et al. [7], Li et al. [6], Yun et al. [24], Zhang et al. [25], and Ai et al. [26], [27] proposed extracting features from other projection domains, such as cubemap, tangent, and icosahedron domains, to complement the ERP domain and reduce the negative effects of panoramic distortions, and introduced an attention module that considers both spatial and angular relations among tokens. Yun et al. [28] presented a panoramic ViT for saliency detection in 360° videos, which leverages a multi-scale feature fusion module and a temporal attention module to capture the spatial and temporal saliency cues. Zhang et al. [29] proposed a method for panoramic semantic segmentation task, which is equipped with deformable patch embedding and deformable MLP modules for handling object deformations and image distortions, and also enhances the mutual prototypical adaptation strategy for unsupervised domain adaptive panoramic segmentation. Recently, multi-task learning for 360° images has gained attention for jointly estimating depth and surface normals. Huang et al. proposed the 360MTL [30], which employs a spherical-aware Vision Transformer [7] to handle distortion and achieved strong performance on both tasks. These works demonstrate the potential and effectiveness of ViTs for 360° vision tasks and motivated us to investigate suitable ViTs tailored for the surface normal dense prediction task, aiming to seamlessly integrate the strengths of the previously mentioned methods. Our objective is to yield robust evaluation results across benchmarks, contributing to further studies in this domain.

3 Our Method↩︎

To address the challenge of estimating surface normals using a single panoramic image for indoor scenes, we propose a novel architecture, PanoNormal. PanoNormal leverages the capabilities of convolutional layers for extracting meaningful embeddings from panoramic images. To address the challenge of spherical distortion in 360° images while maintaining global dependencies, we crafted a U-shaped distortion-aware transformer architecture, drawing inspiration from the approach proposed by Shen et al. [7]. More specifically, we adopt a multi-level structure in our transformer decoder, capturing spatial relationships, processing fine-grained details, and integrating high-level contextual information across different levels. Fig. 2 illustrates the architecture of PanoNormal.

3.1 Network Architecture↩︎

3.1.1 Feature Embedding↩︎

In a conventional ViT-based network, feature embedding undergoes processing through only a single convolutional layer before ViT blocks. It has been demonstrated that this works well for some 360° understanding tasks, such as depth estimation [7]. Nonetheless, although both depth estimation and surface normal tasks demand high-quality contextual information, it is not effective to apply a single convolutional layer to tackle both depth and surface normal estimation tasks. This is primarily due to the intricate nature of these tasks and their feature requirements. Depth estimation benefits from a broader, more global understanding of the scene, which can be achieved with a single convolutional layer. In contrast, surface normal estimation requires capturing detailed local structures and fine patterns, which is better accomplished with multiple convolutional layers. To address this issue, we introduce a series of convolutional layers aimed at extracting more effective features. This approach enhances our model’s ability to gain a more comprehensive understanding of the entire scene, improving performance in surface normal estimation. The feature embedding block consists of three \(3 \times 3\) convolutional layers, each followed by a batch normalization function and a rectifid linear unit (ReLU) activation function.

3.1.2 Distortion-aware Transformer Encoder↩︎

The inherent non-uniform spatial warping of visual features, caused by spherical distortion in the 360° image, is built upon the PanoFormer transformer mechanism [7]. To address issues such as distortions, misalignment, and inaccuracies arising from direct pixel sampling on the ERP image, a tangent projection division strategy is employed. It samples related surrounding pixels of the central pixel (tangent point) on each dense divided plane in the tangent domain, obtaining a series of tokens in patches with decent position representation that can be further transformed to the ERP domain. Subsequently, a conventional multi-head vision transformer block is applied, with the replacement of the feed-forward network to the locally-enhanced feed-forward network [21], [31] for enhancing local feature interaction. The introduced encoder not only computes attention scores between central and correlated tokens but also integrates a trainable token flow. This token flow serves as a bias to adjust the spatial distribution of tangent tokens, supplying the network with additional positional information for learning global dependencies. An example is shown in Fig. 2 (a) and the representation of the self-attention is as follows: \[P(f,\hat{s}) = \sum\nolimits_{m}W_m\left[\sum\nolimits_{(q,k)}A_{mqk} \cdot W'_{m}f(\hat{s}_{mqk}+\Delta s_{mqk})\right]\] where the sampling strategy is adopted on the feature representations \(f\), and is denoted as \(\hat{s}\). The parameters \(m, q, k\) represent the self-attention head, each token, and its neighboring tokens in a tangent patch, respectively. \(W_m\) and \(W'_{m}\) denote the learnable weights of each head, \(A_{mqk}\) is the attention weights for each token, and \(\Delta s_{mqk}\) signifies the learned bias of each token’s query.

The learned embeddings \(P\) are subsequently transmitted through skip links to their respective hierarchical decoder blocks. Simultaneously, they undergo downsampling, reducing the feature size by half while doubling the dimensions, before proceeding to a bottleneck block for decoding.

3.1.3 Multi-level Transformer Decoder↩︎

Our proposed ViT decoder, designed with spherical distortion awareness and a hierarchical structure to predict normal maps of different scales, is illustrated in Fig. 2 (b). The proposed multi-level decoder comprises four independent blocks. Each block processes the concatenation of the upsampled encoded representations \(\hat{f}_{i}\), featuring twice the spatial resolution size and half the number of channels from the preceding level, and the directly propagated features \(\hat{s}_{i}\) from the encoders through the skip-links. The surface normal vectors in various scales are generated as outputs using \(3 \times 3\) convolutions, followed by a hyperbolic tangent (tanh) activation function to constrain them to the range \([-1, 1]\). This process can be described as: \[\hat{\boldsymbol{N}}_i=\tanh\left(\hat{P}(\hat{f}_{i}, \hat{s}_{i})\right)\] where \(\hat{\boldsymbol{N}}_i\) denotes the predicted surface normal maps for the \(i\)-th scale.

This architecture enhances feature analysis across fine and coarse scales, promoting a holistic comprehension of diverse granularities and elevating spatial understanding. It filters noise effectively, preserving essential features and enhancing generalization to unseen data. The significance of the introduced decoder in enhancing final predicted results has been validated through our ablation study (Sec. 4.4).

3.2 Loss Function↩︎

The proposed network generates surface normal maps at multiple scales to capture the scene’s detailed global geometric structure. During training, the predicted maps are upsampled to match the input size using bilinear interpolation. The training process incorporates MSE, quaternion, perceptual, and smooth loss functions, which are detailed as follows:
MSE Loss, \(L_{m}\) is the mean squared error between the ground truth and the predicted normals of each pixel, defined as: \[L_{m} = \sum_{i=1}^{S} \| \hat{\boldsymbol{N}}_i - \boldsymbol{N}_i \|_2\] where \(S\) is the number of scales, and \(N_i\) denotes the ground truth surface normal maps for the \(i\)-th scale.
Quaternion Loss, \(L_{q}\) [10] measures the angular difference between predicted and ground truth normal maps on a pixel-wise basis: \[L_{q} = \sum_{i=1}^{N}\sum_{j=1}^{M} \arctan(\frac{\|\hat{\boldsymbol{N}}_{ij} \times \boldsymbol{N}_{ij}\|}{\hat{\boldsymbol{N}}_{ij} \cdot \boldsymbol{N}_{ij}})\] where \(M\) indicates the number of pixels of the input, and \(j\) indexes the current pixel. Perceptual Loss, \(L_p\) is employed on the finest scales to enhance the generation of finer details: \[L_{p}=l_{\textsubscript{feat}}^{\phi,k}(\hat{\boldsymbol{N}},\boldsymbol{N}) = \sum_{j=1}^{M}\frac{1}{C_{k}M} \| \phi_{k}(\hat{\boldsymbol{N}}_j) - \phi_{k}(\boldsymbol{N}_j) \|_2^{2}\] where \(\phi\) is the VGG-16 network [32] that pretrained on the ImageNet dataset [33], and \(C_{k}\) indicates \(C\) dimensional features for the \(k\)-th layer of the network \(\phi\).
Smooth Loss, \(L_{s}\) quantifies the gradient, \(G\), in the \(x\) and \(y\) directions in the ground truth and the predicted surface normal map at all scales: \[L_{s} = \sum_{i=1}^{N}\sum_{j=1}^{M} (| G_{ij}^{x} |+| G_{ij}^{y} |)\]
The overall loss function of our network is: \[L = \lambda_mL_m + \lambda_qL_q + \lambda_pL_p + \lambda_sL_s\] By default, we set \(\lambda _m = 1.0\), \(\lambda _q = 10.0\), \(\lambda _p = 0.05\), and \(\lambda _s = 0.5\) as the weights for different terms. Our experiments show that this specific combination consistently yields the best results compared to other loss configurations. The effectiveness of each loss function is further validated in our ablation study in Sec. 4.4.2

4 Experiments and Results↩︎

We conducted experimental validation using five widely recognized panorama datasets: 3D60 [34], Stanford2D3D [35], Matterport3D [36], SunCG [37], and Structured3D [38]. These datasets were used for both quantitative and qualitative analysis. Additionally, we included the SUN360 [39] dataset, which contains real-world data without ground truth, for further qualitative and generalization comparisons.

To evaluate the performance of our method, we compared it against HyperSphere [10], which is the current state-of-the-art method specifically designed for 360° surface normal estimation, and 360MTL [30], which predicts 360° surface normals as a sub-task. We also adapted five popular 360° and perspective image depth estimation methods—PanoFormer [7], OmniFusion [6], MonoViT [13], and UniFuse [4]—by modifying their final prediction layer to output three dimensions instead of one, thereby making them applicable to the surface normal estimation task. This adaptation was crucial for validating the effectiveness of our proposed architecture.

4.1 Evaluation Metric and Datasets↩︎

We conducted performance evaluation of surface normal estimation using three standard angular error metrics (mean error (Mean), median error (Median), and mean square error (MSE)), and five accuracy metrics that measure the percentage of pixels where the ratio (\(\delta\)) between the predicted surface normal vector and the ground truth is less than 5°, 7.5°, 11.5°, 22.5°, and 30°. A consistent setting is applied to ensure a fair comparison across all methods, and detailed specifics for each dataset are presented as follows.

3D60 dataset offers a broad spectrum of panoramic images with resolutions of \(256 \times 512\) captured in varied environments. The captured 360° RGB imagery with corresponding information, such as surface normal and depth with specific camera positions are from two real-world indoor scanning environments, Stanford2D3D and Matterport3D, alongside synthetic scenes from the SunCG datasets. The inherent distribution gap among these datasets enhances the model’s generalizability. To facilitate model training and evaluation, we adopt the data splits utilized in HyperSphere, as recommended in the dataset’s documentation. Notably, Matterport3D lacks ground truth data for 360° surface normals, and Stanford2D3D’s surface normal instances lack consistently aligned axes across their data. Consequently, we evaluated them based on specific separations within the 3D60 dataset.

Structured3D is a large-scale synthetic dataset featuring 21,835 RGB images of resolution \(512 \times 1024\). These panoramic images were captured across 3500 scenes, illuminated with cold, normal, and warm lighting. The dataset also includes surface normal, depth, and semantic annotations. We preprocessed and formed the dataset examples with an 8:1:1 ratio, yielding 17,442 training data instances with three distinct lighting conditions (52,326 in total with three lighting conditions) and 2,179 and 2,181 validation and test data instances with randomly selected lighting conditions.

4.2 Implementation Details↩︎

Our experiments were carried out using a single CPU core of an Intel Xeon W-2133 paired with an RTX 3090 GPU, with a batch size configured to 2 and the input resolution set to \(256 \times 512\). The chosen optimizer was Adam, with default settings maintained. The initial learning rate was \(1\times10^{-4}\), decreasing by half every 15 epochs. We trained our model for 110 epochs and incorporated early stopping at the 15^th epoch if there were no further improvements.

4.3 Experimental Results↩︎

We present a comprehensive quantitative comparison between state-of-the-art methods (HyperSphere and 360MTL), algorithms adapted for spherical surface normal estimation, and our PanoNormal model across five datasets, as detailed in Table 1. For a fair evaluation, all models were retrained under identical settings. PanoNormal demonstrates superior performance, achieving state-of-the-art results across all five benchmarks. On average, it delivers a consistent improvement of 6.50% in mean error and 8.96% in MSE, surpassing the previous best methods highlighted in grey.

In specific cases, our model shows significant improvements in the MSE metric: 15.22% on 3D60, 10.94% on Stanford2D3D, 14.81% on Matterport3D, and 25.10% on SunCG. However, the improvement is relatively modest on the Structured3D dataset, with only a 2.69% enhancement in MSE. This less significant difference is due to the complex, synthetic nature of Structured3D scenes, which contain numerous small objects with subtle curvature changes, such as ornaments, drawer handles, and wall-mounted kitchen tools. Additionally, the dataset challenges the models with detailed textures in items like mirrors, glass walls, and carpets.

In addition, PanoNormal shows a marginally higher median error than HyperSphere on the Stanford2D3D dataset, indicating potential sensitivity to outliers. However, this discrepancy also suggests that our method captures finer details. HyperSphere, on the other hand, sacrifices sharp boundaries in its predictions, resulting in more blurred outcomes, as illustrated in Fig. 3.The improvement in \(\delta\) performance underscores PanoNormal’s ability to achieve more accurate predictions, demonstrating its generalizability and effectiveness across diverse domains, including both real-world and synthetic scenarios within the datasets.

We also provide qualitative results, showcasing RGB input images alongside the corresponding predictions generated by various methods, with the ground truth surface normal map shown in Fig. 3. Each dataset’s test example is displayed with specific colors mapped to coordinate information, offering an intuitive representation of surface normals. Areas with unavailable data are denoted by grey on the surface normal map. The PanoNormal model stands out for capturing finer details and sharper boundaries, showing greater sensitivity to subtle surface curvatures.

To assess the generalizability of our method, we conduct surface normal estimation on real-world data, comparing our results with those from HyperSphere and 360MTL. As demonstrated in Fig. 4, PanoNormal produces precise predictions on the unseen real-world SUN360 dataset, accurately estimating the complete boundaries of various objects even in the presence of distortions and complex textures. Noteworthy examples include the bed at the bottom of the ERP image in the first row and the glass door in the third row. In contrast, HyperSphere does not generalize well to real-world panoramic images, further validating the efficacy of our proposed method.

When examining Stanford2D3D and Matterport3D as real-world datasets, it is important to note that their ground truth surface normal maps are constructed with triangle faces, which may introduce certain visual characteristics. However, our method shows a strong ability to generate smooth surface normals with enhanced boundary precision, contributing to a higher standard of visual quality in our results. More discussion are available in the supplementary material.

To evaluate the effectiveness of applying perspective image algorithms to the 360° domain, we conduct experiments using ASNGeo [1], a recent multi-task model designed to predict both depth and surface normal maps for perspective images. ASNGeo includes a depth-to-normal (D2N) layer that calculates surface normals by averaging directions from multiple randomly sampled triangle faces covering each 3D point. We retrained ASNGeo using 360° data, keeping the default settings for the D2N layer to maintain consistency with the original paper. As shown in Fig. 5, the qualitative results from ASNGeo’s surface normal prediction branch reveal suboptimal performance, even when objects in the ERP image exhibit minimal distortion. We also apply the D2N layer to depth maps from both ASNGeo and a recent 360° depth estimation approach (GLPanoDepth [40]). The results show blurry outputs around objects and inconsistent appearances for ceilings and floors. Additionally, deriving surface normals from depth using this method accumulates errors, ultimately leading to poorer results. These highlight the necessity of developing specialized algorithms for 360° surface normal estimation.

4.4 Ablation Study↩︎

width =0.7

4.4.1 Individual component study.↩︎

We conducted an ablation study to validate the key components of our model. As presented in Tab. ¿tbl:tab:ablation95component?, PanoFormer serves as the baseline structure for surface normal prediction. Originally designed for depth estimation, PanoFormer exhibits suboptimal performance when directly applied to surface normals, primarily due to the distinct nature of the tasks. Surface normal estimation relies on fine-grained local geometric details, whereas depth estimation emphasizes global scene structure, highlighting the architectural mismatch when the baseline is used without modification.

To address these limitations we introduce, first, a multi-level decoder, and then add a feature embedding module. The multi-level decoder was particularly impactful, resulting in significant reductions in mean, median, and MSE error metrics by 70.55%, 95.12%, and 80.83%, respectively. We believe that its effectiveness arises from its ability to aggregate multi-scale information, combining low-level features for detailed geometry with high-level contextual cues, which is crucial for accurately predicting surface normals across diverse regions, including areas with sharp boundaries or fine textures. This capability ensures that both global consistency and local precision are maintained, addressing one of the primary shortcomings of the baseline.

Building on this, the feature embedding module was designed to enhance the representation of fine local geometric structures, capturing intricate surface characteristics such as subtle curvature changes and texture discontinuities that are often overlooked in global feature representations. Integrating this module into the architecture led to further improvements, with reductions of 4.32% in the mean error, 16.84% in the median error, and 5.85% in MSE. The module effectively complements the multi-level decoder by focusing on local feature refinement, enabling the model to perceive nuanced geometric variations in complex scenes.

Together, these components address three critical factors: the task-specific mismatch in the baseline architecture, the need for effective multi-scale feature aggregation, and the importance of capturing detailed local geometry. By leveraging these enhancements, our model achieves a significant improvement in surface normal prediction accuracy, demonstrating the value of balancing global context with fine-grained detail. These findings underscore the necessity of architectural adaptations for domain-specific tasks like surface normal estimation and the potential of our approach to set a new benchmark in this field.

width =0.6

width =1

4.4.2 Loss combinations.↩︎

We investigate the effectiveness of various losses on our model’s accuracy to determine the optimal combination for training. As shown in Tab. ¿tbl:tab:ablation95loss?, we began with the common losses: MSE (\(L_{m}\)) and perceptual loss (\(L_{p}\)), on the 3D60 dataset. Adding the quaternion loss (\(L_{q}\)) improved the mean and median error metrics by 2.79% and 47.05%, respectively, though it increased the MSE error, indicating higher sensitivity to outliers. Conversely, adding the smooth loss (\(L_{s}\)) alone decreased performance. However, combining all four losses with specific weights yielded the best results. This suggests that while the quaternion loss sharpens scene details but is sensitive to outliers, incorporating smooth loss helps achieve a balance, enhancing fine-grained details without sensitivity to outliers.

4.4.3 Number of ViT blocks↩︎

The number of ViT blocks affects both the accuracy and efficiency of our method. To investigate this, we progressively replaced pairs of self-attention blocks with convolutional blocks in both the encoder and decoder, resulting in multiple configurations with different ViT-to-convolution ratios (Tab. ¿tbl:tab:ablation95PFblock?). The results show that accuracy consistently decreases as more convolutional layers are substituted, confirming the effectiveness of ViT blocks for capturing long-range dependencies. The best performance is obtained when all blocks are ViTs. Interestingly, the variant “Ours-V3,” which retains self-attention only at the bottleneck and one higher level in both encoder and decoder, achieves competitive accuracy while significantly simplifying the architecture. This indicates that a small number of strategically placed ViT blocks can still preserve most of the benefits of global reasoning, while potentially making the model more suitable for deployment on resource-constrained devices.

4.4.4 Efficiency of our models↩︎

In addition to accuracy, we report computational cost (GFLOPs), model size, and inference speed (frame per second, FPS) in Tab. ¿tbl:tab:ablation95PFblock?. These results highlight a clear trade-off between accuracy and efficiency: ViT-heavy configurations deliver the best error metrics, whereas conv-heavy variants run faster and require fewer resources. This flexibility allows our framework to be adapted to different application requirements, such as prioritizing accuracy in offline high-fidelity settings or emphasizing speed in real-time scenarios. Compared to HyperSphere, our models offer a more favorable balance, achieving comparable or better accuracy with competitive computational efficiency. For instance, mid-range variants like Ours-V3 provide a good compromise, maintaining strong predictive performance while reducing memory usage and improving inference speed, making them attractive candidates for practical deployment.

5 Limitation↩︎

In Tab. 1, our method exhibits a performance of approximately 24.59% higher median error compared to HyperSphere on the Stanford2D3D dataset while demonstrating significant improvements in the metrics of mean and MSE. The enhancements on mean and MSE suggest that, on average, our model provides more accurate predictions. However, the higher median error implies the existence of a subset of predictions with larger errors.

width =0.6

To investigate the factors contributing to the higher median error, we inspected the predicted results. We present three typical cases from the Stanford2D3D dataset, both quantitatively in Tab. ¿tbl:tab:limitation? and qualitatively in Fig. 6. Notably, the ground truth surface normal data in the Stanford2D3D dataset is generated based on the reconstructed triangular mesh of each real-world scene. Predictions aligning with surfaces constructed with triangles may deviate from the authentic appearance of objects. HyperSphere, in such instances, blurs object boundaries and achieves a lower median error on such regions, while our approach predicts a more precise boundary that better aligns with the actual shape of objects. This distinction contributes to the overall higher median error, attributed to the inherent limitations in the quality of the ground truth data.

6 More Comparison on SUN360↩︎

In Fig. 7, we show qualitative comparisons on the SUN360 dataset for 360 surface normal estimation. Our model demonstrates superior performance compared to existing state-of-the-art approaches, includes models such as HyperPSphere, PanoFormer, MonoViT, UniFuse, and OmniFusion, all of which have been evaluated on diverse indoor scenes.

Our model consistently delivers more accurate and coherent surface normal predictions, as evident from the sharpness and continuity of the estimated surfaces across varying room structures. Unlike competing models, which often produce fragmented or blurred normal maps, especially in areas with complex geometries, our approach maintains clarity and precise border of surfaces. This is particularly noticeable in regions with intricate details, such as furniture edges and room corners, where other methods tend to falter.

These results highlight the robustness and effectiveness of our method in handling the challenges posed by real-world 360° imagery. Our method excels in maintaining visual consistency and preserving fine details across various indoor environments, outperforming other models in the process. The ability of our model to produce clear and accurate surface normal estimations, even in complex scenes, sets a new standard for 360 surface normal estimation and establishes it as a reliable solution for applications that demand a high level of environmental understanding.

7 Conclusion↩︎

We present PanoNormal, a deep architecture designed for single 360° surface normal estimation. Our approach leverages the combined strengths of convolutional neural networks for effectively capturing local details and spherical vision transformers for global dependency with distortion awareness. Experiments demonstrate PanoNormal’s significant advances over state-of-the-art methods across five popular 360° panoramic datasets. Additionally, PanoNormal demonstrates enhanced generalizability compared to previous works, with our ablation studies further validating the effectiveness of our proposed architecture. Our findings also reveal that models originally designed for 360° depth estimation are not directly transferable to surface normal prediction, underscoring the necessity of tailored architectural design. This makes it a reliable tool for tasks that demand precise environmental understanding.

Acknowledgements↩︎

This research was supported by the Marsden Fund Council managed by the Royal Society of New Zealand (No. MFP-20-VUW-180) and Tiandi Science and Technology Co. through the Technology Collaboration Project (No. 2025-TD-CXY003).

References↩︎