1 Introduction↩︎

The rapid development of virtual reality technology has opened up new possibilities to create immersive and realistic experiences. With virtual reality headsets, users can enjoy the breathtaking landscapes and unique cultures of various regions of remote travel without leaving the comfort of their home. However, high-quality immersive experiences rely on 3D generated scenes with high realism and consistency, which remains a significant challenge in artificial intelligence and 3D computer vision.

High-fidelity 3D scenes reconstruction utilize 3D reconstruction techniques, including traditional handcrafted [1]–[5] and learning-based reconstruction methods [6]–[9]. The traditional methods relying on scanning technology achieve high-accuracy geometric structures and capture objects and spatial relationships in fine detail. However, they come at the cost of significant time and labor expenses. Moreover, scan-based reconstruction methods often fall short in texture quality, with the generated textures frequently lacking detail and realism. With the rapid advancement of deep learning, many new methods have been introduced for 3D reconstruction with photogrammetry [10]–[13], Neural Radiance Fields (NeRF) [14]–[17], and 3D Gaussian Splatting (3DGS) [18]–[21]. However, these methods heavily depend on multi-view information, which is difficult to obtain in many practical scenarios, thereby limiting their applicability.

To improve the accessibility of multi-view data, several diffusion-based studies have been proposed that iteratively generate images from novel viewpoints for single-view 3D scene reconstruction [22]–[24], partially addressing the limitations of viewpoint availability. However, since only local texture information from existing images is used during the iterative process, the imperfect 3D scene lacks global consistency. This leads to significant visual discontinuities between different regions of the scene, which severely affects both immersion and realism.

Panoramic images, compared to perspective images, offer a broader coverage and richer contextual information, which result in better visual consistency and help address the visual discontinuities. Consequently, some researchers have begun exploring the extensive scene cues in panoramic images to improve the performance of generative models in 3D reconstruction [25]–[27]. We find that high fidelity and availability 3D scenes generation requires the integrity of the scene. However, due to significant foreground-background occlusion, the 3D space generated from panoramic images exhibits noticeable gaps and voids, which negatively affect the immersive visual experience.

Recent works such as WonderWorld [28] and LayerPano3D [29] have highlighted the potential of layered representations for enhancing scene understanding. Building upon these insights, we propose a novel framework named Scene4U, which leverages real-world panoramic images to perform multi-layer 3D scene reconstruction, producing highly realistic and coherent immersive experience scenes.

(1) Following the input of prompt text and the original panoramic image, we first employ a Climate Controller to generate a spatiotemporally specific panoramic image. Subsequently, the generated panoramic image is processed using the open-vocabulary Semantic Segment Anything (SSA) [30] instance segmentation model. Both the segmentation masks and the panoramic image are then fed into a large language model (LLM) for semantic filtering to obtain hierarchical masks.

(2) To elevate the panoramic image to a 360-degree scene representation, we perform layered inpainting based on the multi-layer masks, and employ depth estimation and completion methods to obtain multi-layer repaired scene, ultimately converting the panoramic image representation into a point cloud.

(3) The point cloud from the previous stage is initialized as a 3DGS representation, which is then optimized using a layered training strategy to address scene occlusion issues, resulting in a highly realistic and spatially consistent 3D scene.

Through the multi-stage hierarchical 3D generation, Scene4U enables the conversion of real panoramic images into a high-precision 3D scene, significantly improving the visual quality and consistency of the scene. It provides a more feasible strategy for virtual reality and immersive experience applications. Our main contributions are summarized as follows:

• We propose Scene4U, a layered scene reconstruction framework that generates highly consistent 3D scenes from a single real-world panoramic image. Scene4U effectively addresses challenges such as texture inconsistency and occlusions within complex environments, enabling users to immerse themselves in realistic scene exploration.

• We develop a method that combines instance segmentation with a LLM to achieve effective foreground-background recognition. By leveraging the visual comprehension capabilities of the LLM, the proposed method improves the accuracy of classifying foreground and background regions, thereby facilitating multi-layered 3D scene reconstruction.

• We provide a dataset for panoramic image 3D reconstruction, covering numerous famous landmarks worldwide. The dataset offers users a diverse array of real-world scenes, allowing them to freely explore renowned global attractions from the comfort of their own home in a virtual environment.

2 Related Work↩︎

2.1 Image Variation Based on Diffusion Models↩︎

Image transformation refers to generating diverse style variations based on a given image sample, while preserving the original semantic information and basic visual perception of the image. In recent years, with the substantial advancements in Diffusion Models in the field of image generation, diffusion model-based image transformation methods have been widely applied to tasks such as style transfer [31], [32], novel view synthesis [33]–[35], and image editing [36]–[38]. Limited by the long intervals between data collections, street view data from the same location often presents a monotonous scene environment, which brings challenges to the construction of multi-styled realistic scenes. To address the above challenges, we employed a text-guided image editing approach to temporally initialize the original input scene images, generating the target street-view panoramic images as required by users, with specific details provided in Section 3.1.

2.2 3D Scene Representation↩︎

Traditional 3D scene representations utilize point cloud [39], volume [40], [41], and meshes [42]. While each of these methods has its own advantages, they typically require large amounts of data, leading to high computational costs. Moreover, these methods struggle to meet the high-quality rendering requirements necessary for immersive scenes. With the rapid development of deep learning technologies, implicit representation methods, such as NeRF [7], have demonstrated outstanding capabilities in novel view synthesis and high-quality rendering. However, these methods still face challenges in terms of optimization efficiency and rendering speed. To overcome these limitations, subsequent research has introduced explicit representation techniques on top of implicit methods to enable faster training and rendering [43], [44]. Among these, the 3DGS [8] approach, which is based on Gaussian kernels, achieves real-time rendering and exceptional rendering quality through the use of alpha blending and differentiable rasterization techniques. Therefore, we adopt 3DGS as our method for scene representation in this work.

2.3 3D Scene Generation↩︎

Currently, most of the existing methods depend on multi-view images for 3D scene generation, which infer the 3D structure of a scene from images captured from various perspectives [19], [45]–[47]. However, these methods cannot be directly applied to single-view scene generation. In the field of single-view scene generation, existing methods such as LucidDreamer [48], RealmDreamer [49] and Text2Immersion [50] iteratively refine the scene by moving a virtual camera, gradually generating a complete 3D scene by capturing and inpainting information from different angles. However, during the iterative refinement process, these methods only utilize partial texture information of the scene, neglecting the global consistency, leading to obvious visual discontinuities in the generated 3D scene. In DreamScene360 [51], researchers used panoramic images to build 3D scenes, thereby ensuring the consistency of overall scene information. However, due to insufficient consideration of foreground-background occlusion relationships, DreamScene360 can only allow scene viewing from fixed perspectives and does not support free navigation.

WonderWorld [28] and LayerPano3D [29] have pioneered the concept of distinguishing the spatial relationships among objects in a scene through scene layering.

Inspired by these methods, we employ a semantic-aware layered construction and rendering approach, which removes the restrictions of fixed viewpoints and allows users to navigate within the scene, providing a more immersive experience.

3 Scene4U↩︎

We propose a panoramic image-driven framework for immersive 3D scene reconstruction that removes distracting elements (e.g., pedestrians and vehicles) to render a 3D scene with high visual consistency and scene integrity. The key insight is allowing users to explore scenes from any time. As shown in Fig. 1, Scene4U consists of three main stages. First, a text-driven diffusion model generates a target panoramic image with specific spatiotemporal properties. Next, a large language model assists in decomposing the target panoramic image into multiple layers. These decomposed layers are then processed with image inpainting and depth restoration to obtain complete layered scene information. Finally, we transform the multi-layered panoramic images into a multi-layered 3D scene by 3DGS refinement, creating an immersive scene.

3.1 Climate Controller↩︎

To address the limited scene environmental conditions in real-world panoromatic images, we first introduce a text-guided Climate Controller based on Instruct Pix2Pix [52]. The climate controller applies environmental condition constraints to the input panoramic images, generating corresponding images in the target domain by specifying conditions such as season, time of day (e.g., daytime or nighttime), etc. These generated images are then used as input for the subsequent layered panoramic reconstruction. As shown in Fig. 2, the Climate Controller enables the synthesis of diverse scene environments, producing images with various times and weather conditions.

3.2 Layered Panorama Construction↩︎

Complex dynamic foreground objects in real-world scenes cause occlusion, making parts of the background invisible. To this end, we propose a novel layered representation method for 3D reconstruction, which eliminates the invisible background and constructs a complete 3D scene by dividing it into multiple visual layers.

Semantic-Aware Layer Parsing. Multi-layer representation requires accurate identification of hierarchical objects in the scene. LayerPano3D[29] innovatively proposes scene layering method using instance segmentation prior. Instance segmentation methods are generally categorized into close-vocabulary [53] and open-vocabulary [30] approaches. As illustrated in Fig. 3, the former is often constrained by a predefined set of categories, which limits its adaptability to the complexity of real-world scenarios. To address this, we propose a novel approach that effectively resolves the applicability issues of existing methods in complex environments.

Firstly, we predefine hierarchical category representations of the scene, including sky \(L_{sky}\), background \(L_{background}\), foreground \(L_{foreground}\) and dynamic object \(L_{dynamic}\) layer, which are sequentially numbered as 3, 2, 1, and 0, respectively. Secondly, we employ a fine-grained open-vocabulary instance segmentation [30] to extract individual semantic objects from the panoramic image. We observe that the predicted instances are difficult to directly categorize into hierarchical layers based on semantic information, especially for class-agnostic masks. Therefore, we utilize LLM [54] to interactively perform semantic filtering and hierarchical classification of objects. Specifically, we input the original RGB image and corresponding segmentation masks into the LLM and use predefined hierarchical categories for prompting. The LLM then outputs the hierarchically filtered segmentation masks. Finally, we obtain multi-layer segmentation masks with spatial hierarchy, as shown in Fig. 4. Notably, we define objects that are likely disappear in the short term as the dynamic objects. Removing dynamic objects helps generate immersive scenes that are more spacious and natural. While there are challenges in the definition of categories, the segmentation results (Fig. 4) are robust to occlusion issues and geometric orderings.

Occlusion-Guided Layer Recovery. The occlusion presents an inevitable integrity issue in layered image for 3D scene construction. To address this, we use the FLUX-inpainting to progressively repair each layer using the hierarchical masks. Based on the given mask (e.g., the areas to be restored), we utilize image context information to render and fill in the matching pixels. Layer-by-layer repair ensures semantic consistency across layers while restoring occluded content, resulting in high-quality generated scene. For instance, the foreground mask is used to help the repair of occluded areas in the background, while the background mask, together with the foreground mask, assists in repairing of occluded sky regions. During the repair process, we use ‘no objects present’ as a prompt to guide the repair model, allowing it to seamlessly transition and restore the original details of the scene when filling in the content of each layer. This process is formalized as: \[\tilde{I}_{l+1} = Inpaint_{RGB}(\tilde{I}_{l}, M_{l+1}),\] where \(Inpaint_{RGB}\) is the function of FLUX-inpainting, \(\tilde{I}_{l}\) and \(M_{l+1}\) denote the RGB panoramic image of the \(l\)-th layer and the mask for the \((l+1)\)-th layer, respectively. The repaired image, obtained after filling the missing regions, maintains both structural and textural consistency as well as completeness.

Hierarchical Spatial Inference. To construct a complete and spatially accurate 3D scene, we perform depth estimation and depth completion on the layered images to ensure spatial consistency across them. Inspired by LayerPano3D [29], we utilize 360MonoDepth [55] to project the panorama image onto 20 overlapping perspective patches and use a pretrained monocular depth model ZoeDepth [56] to calculate depth for each projection. To address the affine ambiguities in scale and displacement, we align each projection by inputting the optimized parameters into a multi-scale and spatially varying deformation field, to get a high-resolution panoramic depth map. We use the existing depth information and RGB texture to predict and complete the masked regions without depth value, mathematically expressed as follows: \[\tilde{D}_{l} = Inpaint_{depth}(\tilde{I}_{l}, D_{l+1}, M_{l}), \label{depth-completion}\tag{1}\] where \(Inpaint_{depth}\) denotes the model for depth completion. \(D_{l+1}\) represents the depth completion result for the \((l+1)\)-th layer, \(\tilde{I}_l\) is the RGB panorama of the \(l\)-th layer, and \(M_{l}\) denotes the mask of the \(l\)-th layer, respectively. Notably, to ensure that the spatial relationship between the depth information of the background and sky layers remains consistent with that of the foreground layer, we perform depth estimation exclusively for the foreground layer, while using depth completion [57] for the background and sky layers.

3.3 Multi-layer Panoramas to 3D Scene↩︎

Panorama to Point Cloud. The restored panoramic image is converted into a point cloud to construct a 2D-3D representation. Using the equidistant projections of the sphere, we efficiently convert the 2D panorama into 3D point cloud without additional computational overhead. Specifically, for each pixel located at \((i, j)\) in a panoramic image \(\tilde{I} \in \mathbb{R}^{H \times W}\), we calculate its latitude angle \(\theta\) and longitude angle \(\phi\) using Eq. 2 . Then, we derive the corresponding 3D coordinates \((X, Y, Z)\) for each pixel \((i, j)\) based on its latitude, longitude, and depth values, as Eq. 3 .

\[\theta_{i,j} = \frac{\pi i}{H}, \quad \phi_{i,j} = -\frac{2\pi j}{W} + \pi, \label{pixel2angle}\tag{2}\] \[\begin{pmatrix} X \\ Y \\ Z \end{pmatrix} = d_{(i,j)} \times \begin{pmatrix} cos\phi_{i,j} \cdot cos\theta_{i,j} \\ cos\theta_{i,j} \\ sin\theta_{i,j} \cdot cos\phi_{i,j} \end{pmatrix} \label{pixel2point}\tag{3}\] Following on the above formula, we sequentially calculate the 3D coordinates of each pixel in the panorama, forming a point cloud that integrates both visual texture and spatial structure. To ensure the accuracy of the multi-layer structure and the independence of point cloud, we only convert the pixels within the mask after filtering pixels in each layer. Finally, We initialize the multi-layer point cloud as 3DGS strictly following the layer index.

Layered Numbered Optimization. 3DGS is able to accurately present the 3D representation of a scene captured from multi-view images. Unsatisfactorily, rendered 3DGS panorama results in distortion due to the significant differences in projection form and viewpoint characteristics between panoramic and perspective views. Unlike previous methods, we decompose and transform the panorama into standard perspective views from different angles through camera field-of-view observations. We set up a set of cameras to cover the panoramic area for viewpoint sampling, where each camera shares the same intrinsic matrix but has its own independent extrinsic matrix. This setup ensures that the camera group observes different regions of the 3D space with consistent viewpoint parameters. Following Eq. 4 , we capture the perspective views after sampling the panorama and use them as the ground truth for that viewpoint: \[\begin{align} x_e &= \frac{x \cdot FOV_x + 2\theta_0 + 2\pi}{4\pi} \cdot W_e, \\ y_e &= \frac{y \cdot FOV_y + 2\phi_0 + \pi}{4\pi} \cdot H_e \label{E2P} \end{align}\tag{4}\] where \(x_e\) and \(y_e\) represent the pixel coordinates in the panorama, \(x\) and \(y\) are the normalized coordinates in the perspective image, \(FOV_x\) and \(FOV_y\) denote the horizontal and vertical field of view, \(\theta_0\) and \(\phi_0\) are the center angles for longitude and latitude, and \(W_e\) and \(H_e\) indicate the width and height of the panorama, respectively.

We find that occlusion effects lead to inconsistencies in the layered scene. In response to the above difficulties, LayerPano3D [29] took the lead in proposing a solution by optimizing the 3DGS model layer by layer in multi-layer panoramas using foreground asset masks to achieve scene refinement. Followed by LayerPano3D [29], we perform independent optimization for each layer during the 3DGS refinement. Due to the clear hierarchical structure of the scene, where the back layers do not occlude the front layers, starting the optimization from the back layers helps to establish foundational background information and ensures depth consistency across layers.Therefore, We extract the 3DGS of specific layers in depth order (from the back layers to the front) for optimization. In the training phase, we set the loss function as a weighted sum of \(\mathcal{L}_{1}\) and \(\mathcal{L}_{D-SSIM}\): \[\begin{align} \mathcal{L} = &(1 - \lambda)\mathcal{L}_{1}(I_{render}^{l}, I_{gt}^{l} \odot M_{l}) + \\ &\lambda \mathcal{L}_{D-SSIM}(I_{render}^{l}, I_{gt}^{l} \odot M_{l}), \end{align} \label{loss-cal}\tag{5}\] where the hyperparameter \(\lambda\) is set to 0.2. \(\odot\) denotes the intersection between ground truth image and layered mask, to prevent the pixel value from other regions from interfering with the loss function. Moreover, the optimization process is performed separately for each layer, masking the loss function gradients of other layers to prevent affecting the parameters of those layers.

4 Experiments↩︎

4.1 WorldVista3D Dataset↩︎

To provide a diverse immersive experience of real-world scenes and to verify the robustness of the Scene4U, we build a 3D real-world scene reconstruction dataset of famous landmarks–WorldVista3D, with 120 panoramic images of well-known tourist attractions worldwide, obtained from the Google Street View API . All image resolutions in WorldVista3D are resampled to a uniform scale of \(2,048 \times 1,024\). Note that this resolution is also used as the input scale for training our model of Layered Panorama Reconstruction.

4.2 Implementations Details↩︎

In the Layered Panorama Reconstruction stage, we first resample the input panoramic images to a resolution of \(2,048 \times 1,024\). Then, we apply the Semantic Segment Anything model to generate initial instance segmentation masks. Using QWen-Plus-Latest, we differentiate segmentation labels and produce corresponding masks for the multi-layer panorama. The FLUX.1 model is used for image inpainting to obtain repaired panoramic results for each layer. Finally, the 360monodepth and depth-inpainting models are employed for depth estimation and restoration.

At the Multi-layer Panorama-to-3D Scene stage, we generate an initial scene point cloud along the camera’s ray direction based on the depth map, and assign a 3DGS index to each layer for subsequent layered optimization. During the 3DGS refinement stage, we independently train the Gaussian Spheres for each layer according to the assigned indices, utilizing supervised perspective images at a resolution of \(512 \times 512\). For the training setup of the Gaussian points in each layer, we set the number of iterations to \(3,000\), \(4,000\), and \(3,000\) for the sky, background, and foreground layers, respectively, while disabling the splitting and pruning operations of the 3DGS. All experiments are conducted on an NVIDIA A100 80G GPU.

4.3 Evaluation Metrics↩︎

To comprehensively evaluate the performance of Scene4U in reconstruction quality and rendering robustness, we employ classic reconstruction metrics alongside no-reference image quality assessment metrics. Specifically, to assess the reconstruction quality of the scene, we select the Structural Similarity Index Measure (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Learned Perceptual Image Patch Similarity (LPIPS) [58] to calculate the similarity between rendered image and reference image. Additionally, to evaluate the robustness of the rendering results, we use traditional image quality assessment metrics, including the Natural Image Quality Evaluator (NIQE) [59] and the Blind / Referenceless Image Spatial Quality Evaluator (BRISQUE) [60], to assess the quality of no-reference images.

4.4 Comparisons with Other Methods↩︎

We select publicly available methods, LucidDreamer [48] and DreamScene360 [51], as baselines for performance comparison with our proposed Scene4U. LucidDreamer takes a single image and textual prompts as input, employing a progressive inpainting strategy to generate a 360-degree scene. This method incrementally fills and expands the image content through multiple iterations to create a complete view. Since the original LucidDreamer does not support panoramic image input, we modify it to accept an input panorama as the initial scene to ensure a fair evaluation of the baseline methods. In contrast, DreamScene360 generates a 360-degree panoramic 3DGS scene based on text prompts, converting textual descriptions into a complete and high-quality 3D scene. To ensure a fair comparison, we excluded the Climate Controller module from the comparative experiments and additionally computed the metrics without removing dynamic objects.

Fig. 5 presents the qualitative comparison results of different methods. Our method achieves the best performance in five metrics compared to previous state-of-the-art methods. Scene4U surpasses DreamScene360 in scene details, achieving finer reconstruction results. As shown in Tab. 1, Scene4U achieves improvements of 7.79% and 24.24% in PSNR and LPIPS, respectively. The quantitative results demonstrate the stability and robustness of the proposed method in panoramic image reconstruction. Moreover, the proposed method demonstrates optimal training efficiency, proving the lightweight nature of layered reconstruction.

1mm

4.5 Ablation Study↩︎

The ablation study on the Layered Panorama Reconstruction is presented in Fig. 6. Compared to the single-layer panorama reconstruction, the generated scene reveals background areas that are occluded by the foreground objects from new perspectives when the user moves within the scene. Overall, layered reconstruction effectively addresses the voids caused by foreground-background occlusions. The quantitative analysis results of different configurations under continuous movement in a specified direction within the scene are presented in Tab. 2. The results demonstrate the proposed Layered Panorama Reconstruction strategy improves robustness over longer distances, achieving 5.93% and 3.35% improvements in NIQE and BRISQUE against the single-layer panorama scene after moving 0.3 m, respectively. These results confirm that our generated realistic 3D scenes meet user demands for unrestricted navigation.

5 Conclusion↩︎

In this work, we propose a novel and effective framework, named Scene4U, for generating highly realistic and consistent 3D scenes through a hierarchical reconstruction strategy. First, we stylize the input panorama and perform multi-layer decomposition with the assistance of an open-vocabulary segmentor and LLMs. Subsequently, we remove dynamic and foreground objects from the scene and design layered inpainting of the occluded areas based on image texture and depth information. Finally, the multi-layer panorama is initialized as a 3DGS representation and hierarchically refined to construct immersive scenes with semantic and structural consistency. Comprehensive experimental results demonstrate that Scene4U outperforms previous methods in terms of visual quality and achieves a realistic visual experience, especially after the removal of dynamic object layers. Scene4U is also competitive in efficiency. Overall, our Scene4U provides a novel solution for reconstructing multi-temporal, dynamic-object-free, globally consistent, and explorable immersive 3D scenes from panoramic images.

References↩︎