1 Introduction↩︎

Large-scale 3D reconstruction from UAV imagery has become essential for urban planning, infrastructure monitoring, and digital twin applications. The reconstruction task is even more challenging in complex cities in developing countries like India. Current methods span various approaches: multi-view stereo [1], neural radiance fields [2], and point-based representations, but 3D Gaussian Splatting [3] has emerged as the leading technique. Its explicit primitive representation enables fast training and real-time rendering while achieving photorealistic quality. Urban reconstruction and digital twin creation particularly benefits from this efficiency when processing hundreds of UAV images covering vast area.

Recent large-scale 3D reconstruction methods CityGaussianV2 [4], VastGaussian [5], and GS4Buildings [6] handle massive urban scenes efficiently. To simulate autonomous systems and urban planning, generating building twins with accurate facades has become necessary, as current Gaussian-splatting-based methods cannot reproduce accurate 3D geometries. This occurs because Gaussian primitives’ smooth spatial transitions through their continuous falloff functions. Standard optimization minimizes global photometric error without explicit constraints at boundary geometry. The 3D models look acceptable from a distance but fail under close inspection, building facades blur into surroundings, window patterns lose definition, roof corners become rounded.

Recent work explores edge awareness in 3D reconstruction. Methods like DET-GS [7] and DN-Splatter [8] use depth maps to guide where Gaussians are placed, but depth estimation from UAV imagery often fails. Wide camera baselines cause correspondence problems, repetitive building textures confuse matching algorithms, and computational costs explode with scene size. EdgeGaussians [9] introduces specialized primitives for boundaries, requiring substantial changes to the reconstruction pipeline. These approaches either break down on large scenes or need too much engineering for modest improvements in 3D building twin quality.

We take a different path. Rather than modifying the 3D representation or relying on unreliable depth, we supervise the 3D reconstruction process using semantic boundary information. From SAM3 [10], we generate binary building masks from UAV images using text prompts. We apply edge detection to the original images, then use these masks to isolate edges that belong to buildings roofs, facades, and corners. During 3D Gaussian Splatting optimization, we supervise the training such that when rendered from any viewpoint, these building edges appear sharp. This forces the Gaussians to arrange themselves in 3D space to produce crisp boundaries. The primitives learn positions and shapes that create well-defined building geometry rather than smooth, blurred transitions.

Our contributions are:

• A method to accurately extract building edges from UAV imagery using SAM3 semantic segmentation combined with image edge detection, isolating architecturally meaningful boundaries.

• An edge regularization loss integrated into 3DGS optimization that supervises 3D Gaussian arrangement to produce sharp building edges in the reconstructed 3D model, requiring no architectural modifications.

• Validation across diverse scenes like residential buildings, commercial structures, university campuses, and dense urban areas with varied roof geometries and architectural styles, demonstrating our method’s generalization capability.

2 Related Work↩︎

2.1 3D Gaussian Splatting↩︎

3D Gaussian Splatting [3] introduced an efficient alternative to NeRF [2] for novel view synthesis. The method represents scenes as 3D Gaussian primitives with learnable parameters (position, covariance, opacity, color). Differentiable rasterization enables fast training and real-time rendering. Extensions have improved various aspects: Mip-Splatting [11] that addresses aliasing, 2D Gaussian Splatting [12] using planar primitives for better surfaces reconstruction. For large-scale scenes, CityGaussian [13] and VastGaussian [5] partition space to handle massive datasets. These methods focus on geometric accuracy, rendering speed, or scale. None specifically addresses edge sharpness in 3D appearance.

2.2 Reconstructing digital twins of buildings↩︎

Building reconstruction from UAV images presents unique challenges in preserving accurate building facades in digital twins. GS4Buildings [6] adapts 3DGS for building-centric reconstruction with geometric regularization. PG-SAG [14] uses planar-based supervision on surface normals to improve wall flatness. SF-Recon [15] tackles large-scale urban scenes with surface-aware primitives. These approaches constrain geometry, ensuring walls are flat, roofs are planar. Our work is complementary: we constrain appearance at edges regardless of underlying geometry.

2.3 Edge-Aware Reconstruction↩︎

Recent 3DGS extensions explore edge awareness in reconstruction. DET-GS [7] and DN-splatter [8] use depth and normal estimates to guide Gaussian densification for better 3D boundary geometry. However, depth estimation proves unreliable for UAV images due to wide baselines and texture ambiguities. EdgeGaussians [9] introduces specialized edge primitives for 3D boundary representation, requiring substantial architectural changes to the splatting framework. These methods focus on where to place Gaussians in 3D space. We focus on how those Gaussians should be configured to produce sharp edges when the 3D model gets rendered from any viewpoint, directly supervising the reconstruction process rather than the primitive placement strategy.

2.4 SAM in 3D Reconstruction↩︎

The Segment Anything Model [16] has found applications in 3D tasks. SAGA [17] and Gaussian Grouping [18] use SAM masks to learn 3D semantic features for object manipulation. SA-GS [19] applies masks for spatial regularization, controlling how Gaussians distribute within segmented regions. These works use SAM for identifying objects in 3D scenes. We use SAM3 [10] with text prompts generate binary building masks, we apply edge detection to training images and use masks to isolate building edges, then supervise 3D reconstruction so these edges appear sharp in the final model. SAM3 provides semantic context about what constitutes a building, while our supervision targets achieving sharp 3D geometry at those building boundaries.

3 Method↩︎

Our approach integrates semantic edge supervision into standard 3DGS training without modifying the underlying architecture. The method consists of three components: SAM-based edge extraction, rendered edge computation, and edge alignment loss.

3.1 SAM-Based Edge Extraction↩︎

We employ SAM3 [10], a concept-based segmentation model that extends the Segment Anything Model [16] with text-guided capabilities. Unlike SAM’s point/box prompts, SAM3 accepts natural language descriptions, enabling precise building detection through descriptive text prompts. For each training image, we apply SAM3 with building-specific text prompts (“building”, “concrete structure”, “modern architecture”, “white building facade”, “roof”) to segment building regions comprehensively. The resulting binary masks \(M \in \{0,1\}^{H \times W}\) delineate building boundaries with high precision.
To extract edge information, we apply Sobel filtering to original training images rather than binary masks, preserving fine-grained features. Sobel outputs continuous gradient magnitudes which is differentiable for backpropagation and runs efficiently as Conv2d operations. After bilateral filtering for noise reduction, Sobel computation follows: \[G_x = \begin{bmatrix} -1 & 0 & 1 \\ -2 & 0 & 2 \\ -1 & 0 & 1 \end{bmatrix} * I_{\text{smooth}}, \quad G_y = \begin{bmatrix} -1 & -2 & -1 \\ 0 & 0 & 0 \\ 1 & 2 & 1 \end{bmatrix} * I_{\text{smooth}}\]

The edge magnitude is computed as \(E_{\text{image}} = \sqrt{G_x^2 + G_y^2}\). We then apply semantic filtering using the SAM3 derived building mask: \[E_{\text{SAM}} = E_{\text{image}} \odot M\]

where \(\odot\) denotes element-wise multiplication. We apply slight morphological dilation (5×5 kernel) to \(M\) before masking to ensure edge coverage near boundaries. This produces semantically-aware edges capturing architectural features.

3.2 Rendered Edge Computation↩︎

During training, we render RGB images \(I_{\text{render}}\) through differentiable gaussian rasterizer. We compute edge maps from rendered images using the same Sobel process to ensure consistency. After converting to grayscale, we apply Sobel filtering directly without bilateral smoothing, as rendered images are already smooth by construction: \[E_{\text{render}} = \sqrt{(G_x * I_{\text{render,gray}})^2 + (G_y * I_{\text{render,gray}})^2}\]

We normalize both \(E_{\text{render}}\) and \(E_{\text{SAM}}\) to [0,1] range to ensure balanced comparison.

3.3 Edge Alignment Loss↩︎

We formulate edge supervision as an L1 distance: \[\mathcal{L}_{\text{edge}} = \frac{1}{HW} \sum_{i,j} |E_{\text{render}}(i,j) - E_{\text{SAM}}(i,j)|\]

This loss integrates into standard 3DGS training through weighted combination: \[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{RGB}} + \lambda_{\text{SSIM}} \mathcal{L}_{\text{SSIM}} + \lambda_{\text{edge}} \mathcal{L}_{\text{edge}}\]

Edge loss computation executes efficiently through Conv2d operations with fixed 3×3 Sobel kernels. SAM3 edge maps are precomputed offline and loaded during training without gradient tracking, ensuring memory efficiency.

4 Experiments and Results↩︎

4.1 Experimental Setup↩︎

For training and evaluation we are using two UAV datasets, UrbanScene3D [20] which has scenes like PolyTech and Art Sci, and Gehukheda area of Bhopal which contains self-collected residential and commercial building imagery with diverse roof geometries. PolyTech (763 images) covers university campus buildings with varied architecture. Art Sci provides dense urban layouts with complex occlusions. All scenes processed at resolution factor 1/2.

We test at two training lengths: 7K and 15K iterations. This shows both convergence behavior and practical efficiency. Standard 3DGS parameters with adaptive density control, exponential learning rate decay, and \(\lambda_{\text{SSIM}} = 0.2\). Camera poses from COLMAP [1]. SAM3 edge maps precomputed via text-prompted segmentation and Sobel filtering.

4.2 Results↩︎

Tables 1 and 2 show quantitative results. Edge supervision never degrades baseline performance improvements occur consistently across metrics and datasets. Memory requirements unchanged since edge maps load without gradients.

Comparison with 2DGS (a planar-primitive method) reveals complementary behavior. While 2DGS improves geometric accuracy through explicit surfaces, it doesn’t address RGB edge appearance. Our method targets visual sharpness directly, suggesting these approaches could combine in future work. SuGaR similarly focuses on mesh extraction rather than edge rendering quality.

5 Conclusion↩︎

We introduced semantic edge supervision for 3DGS building reconstruction using SAM3 boundaries as guidance. The approach addresses a specific problem of edge blurriness in 3D reconstructed buildings. By supervising RGB gradients at semantically meaningful locations, we improve visual quality without architectural modifications to 3DGS.

Limitations of this method depends on SAM3 segmentation accuracy, and \(\lambda\) requires per-scene tuning (though values cluster around 0.001-0.1).

The method adapts to different scene complexities through \(\lambda\) tuning and maintains improvements even with reduced training time.

Our work is complementary to geometric methods like 2DGS and PG-SAG that constrain surfaces through depth or normals. We constrain appearance. Future work could combine both approaches, using geometric methods for structural accuracy and appearance based edge supervision for visual sharpness. The edge alignment framework could extend to other Gaussian-based reconstruction methods beyond 3DGS.

References↩︎