1 Introduction↩︎

FEW-VIEW surface reconstruction, which recovers the scene’s visible surfaces from a few posed RGB images, provides the essential 3D modeling required for various applications including autonomous exploration, augmented reality, and industrial inspection. In these applications, robots need to build the scene online as they travel and quickly decide how to act based on its structure. The throughput and geometric accuracy of the reconstruction algorithm are therefore critical for robots’ fast and safe interaction with the environment.

As these tasks usually run on platforms with limited on-board compute and battery, such as small drones and wearable devices, surface reconstruction algorithms must also maintain a small memory footprint (the maximum memory in use during execution) to be effective. Beyond competing for a fixed capacity with other concurrent tasks such as localization, a high-memory reconstruction algorithm must frequently transmit data to and from the off-chip DRAM, which could dominate the system compute energy [3] and introduce delays to prevent full utilization of the compute units [4]. Therefore, surface reconstruction algorithms must also limit their memory footprint to support long-term and efficient operation.

Recently, 3D Gaussian Splatting (3DGS) [5]–[8] has emerged as a compelling candidate scene representation due to its high fidelity and efficient view recovery through rendering. However, quickly and accurately reconstructing such a representation from a few views with a low memory footprint remains challenging because few-view reconstruction is fundamentally ill-posed and requires additional compute and memory to resolve the ambiguity. From only a few views, many distinct 3D surfaces reproduce almost identical images, which makes it difficult to infer the true ones from the captured images alone. As a result, traditional 3DGS [5]–[8], which fits Gaussian splats by matching their rendering to the captured images, struggles with few views by producing “floater” artifacts and erroneous depth [9]–[11].

To resolve ambiguity, an emerging family of end-to-end methods [12]–[16] leverages neural networks to directly regress the splat parameters from input images. Trained on various scenes, the networks semantically understand the scene structure and assign plausible positions to each splat with rules learned from data. However, encoding large volumes of prior knowledge on 3D scene structures typically requires heavy backbones such as Vision Transformers (ViTs), which need hundreds of MBs to tens of GBs of memory for weights and activations and take up to seconds to complete a single inference on high-end desktop GPUs. Moreover, the networks are usually overfit to the scale, texture, and camera configurations of the training set and predict erroneous depth when test views differ in those aspects. This prohibitive resource usage and poor generalization, resulting from attacking the whole ill-posed problem with a monolithic network, has limited their potential for on-board reconstruction.

In this letter, we resolve these issues by exploiting a well-posed subproblem within the otherwise ill-posed task: given cross-view 2D splat correspondences, 3D splats analytically follow from classical multi-view geometry. Unlike a deep network approximating the same transformation, triangulation solves 3D splats from the projective constraints of their own correspondences rather than recalling prior knowledge learned from other scenes; costing a few linear algebra operations per splat, it recovers the metric scale and automatically adapts to varying cameras. Identifying corresponding splats remains ill-posed and benefits from a neural prior, but the networks can stay lightweight while remaining generalizable: they only compare images to track 2D texture and need no capacity to memorize 3D structure or approximate multi-view geometry.

Therefore, we propose G\(^2\)SR, an efficient Gaussian-based few-view surface reconstruction framework with a small neural frontend and an analytic backend for solving subproblems of matching posedness (1, middle):

• (Frontend, ill-posed) a splat detection network partitions an input image into boundary-respecting 2D Gaussian splats; an affine tracking algorithm then estimates their correspondences across views from neural optical flow;

• (Backend, well-posed) a geometric triangulator recovers 3D splats by matching projections to correspondences under the Hellinger distance using Gauss–Newton iterations.

On the ScanNet [2], Replica [1], and DTU [17] datasets, G\(^2\)SR matches or exceeds the depth-estimation and mesh reconstruction accuracy of state-of-the-art (SOTA) end-to-end methods in 2- and 3-view reconstruction. Consistent with this design, G\(^2\)SR recovers depth at metric scale without needing post-hoc alignment to metric truth ([sec:exp:geom-acc:depth-estimation]), and its accuracy holds when the input camera geometry varies across datasets ([sec:exp:geom-acc:generalization]). When measured on a single NVIDIA RTX 4090 GPU at a resolution of \(384 \times 512\), G\(^2\)SR operates at – RPS while only consuming – MB of memory, resulting in a speedup and memory reduction of up to \(\times\) and \(\times\), respectively (1, bottom).

2 Related Work↩︎

2.1 End-to-end GS↩︎

Traditional per-scene 3D GS [5], [8] fits the Gaussian parameters to each scene via iterative differentiable rendering. Despite recovering photorealistic images at thousands of FPS [5], the reconstruction takes up to 30,000 optimization steps and hours per scene, and is susceptible to the “floater” artifacts due to view ambiguity in few-view settings [16]. Both combined make these methods unsuitable for online applications.

The recent end-to-end Gaussian Splatting has opened new possibilities for near-instantaneous few-view 3D reconstruction. By using deep neural networks to predict splat attributes decoded from unprojected per-pixel depths [12], [14], [18]–[20] or point clouds [15], [19], [21]–[23], these methods recover the 3D scene structure by leveraging prior knowledge learned from various training scenes.

However, locating and shaping Gaussians in 3D forces the networks to both encode scene priors that resolve the ambiguity and aggregate cross-view information to approximate multi-view geometry, driving current SOTA frameworks to employ “heavy” backbones, primarily Vision Transformers (ViTs), that incur significant memory and compute yet are hard to generalize to untrained scenes. CNN-based variants such as pixelSplat [24] and MonoSplat [14], despite being less resource-demanding, degrade in geometric accuracy due to smaller network capacity. In contrast, G\(^2\)SR uses an analytic backend to solve the well-posed triangulation subproblem from multi-view constraints, while lightweight networks supply only 2D image-plane evidence, therefore reducing compute and memory while ensuring across-scene generalization.

2.2 Multi-view Geometry in GS↩︎

In both per-scene and end-to-end Gaussian splatting, multi-view geometry has so far served only auxiliary roles of initialization, regularization, or post-processing. Per-scene methods usually seed the splat centers with sparse point clouds from offline SfM [5]. More recent variants, such as EDGS [25] and TriaGS [26], use dense triangulation to replace densification or as consensus constraints to condition the optimization under dense views. To improve surface smoothness and mitigate the “floater” artifacts in few-view settings, subsequent works introduce geometric priors such as monocular depth and normal consistency to regularize the optimization [8]–[10], [27]. However, even with the constraints imposed, the splats are still recovered using photometric optimization in each scene, whose long latency precludes their online use. In the end-to-end regime, multi-view constraints are often implicitly integrated within the neural architecture via feature cross-attention in 3D cost volumes [12], [18], [20], [24] or among the transformer tokens, or explicitly used to prune floater Gaussians after initial inference [13], [18]. In contrast, our framework is among the first to employ analytic multi-view triangulation as the primary 3D solver in a feed-forward pipeline, rather than as an auxiliary for seeding or regularizing per-scene optimization [25], [26] or for pruning after network inference [13], [18], delivering online, metric-scale reconstruction within hundreds of megabytes of memory.

3 Proposed Method↩︎

In this section, we present G\(^2\)SR, an efficient Gaussian-based few-view surface reconstruction framework that analytically solves the well-posed triangulation subproblem of the task and learns the ill-posed splat detection and tracking subproblems. Our design rests on a simple observation: a Gaussian splat lying on the scene surface is fully determined by its 2D projections across the input views, thus making its position, shape, and orientation recoverable through multi-view geometry. G\(^2\)SR mirrors the detect–track–optimize pipeline of feature-based SLAM (2): a lightweight network detects 2D splats on the reference view (3.2) from local texture; an optical-flow-based sigma-point warping algorithm tracks each splat across the remaining views as an affine correspondence (3.3); and a Gauss–Newton solver triangulates every 3D splat from these correspondences using a few linear algebraic operations (3.4), so that geometric accuracy and generalization rest on a multi-view-constrained-backend, making lightweight networks suffice in the frontend. Before detailing each stage, we first introduce notations and formalize the projection-matching objective the backend solves (3.1).

3.1 Formulation↩︎

Given \(K\) posed RGB images \(I_{1\dots K}\) of a static scene, where \(K = 2, 3\) in the few-view setting we target, we seek to recover a set of \(N\) oriented 3D Gaussian splats to represent the visible surface. Similar to 3DGS [5], we denote the geometry of each splat as \(\mathcal{G} = (\boldsymbol{\mu}, \Sigma) = (\boldsymbol{\mu}, R S S^T R^T)\), which is parameterized by mean \(\boldsymbol{\mu} \in \mathbb{R}^3\), rotation \(R \in \mathrm{SO}(3)\), and scales \(S = \mathop{\mathrm{diag}}(s_u, s_v, \varepsilon)\) with \(s_u, s_v > 0\) and a fixed small positive \(\varepsilon\). Together with view-independent color² \(\boldsymbol{c} \in [0, 1]^{3}\) and opacity \(\alpha \in [0, 1]\), the splat represents a colored, semi-transparent, thin elliptical disk centered at \(\boldsymbol{\mu}\) with semi-axes \(s_u, s_v\) and normal \(R\boldsymbol{e}_z\); each splat thus models a local planar patch of the scene surface, i.e., a surfel.

During reconstruction, we recover the 3D splats through projection matching: rather than rendering them and comparing with the captured pixels, we match each splat’s 2D projection in every view against a per-view 2D target. Specifically, let \(\mathcal{G}_i^*\) denote the splats that, once rendered, reproduce each captured image by minimizing the 3DGS rendering objective \[\label{eqn:conventional-gs-opt} \left\{ \hat{\mathcal{G}}_i \right\} = \mathop{\mathrm{arg\,min}}_{\left\{ \mathcal{G}_i \right\}} \sum_{k \in [K]} \mathcal{L} \left( \mathcal{R} \left( \left\{ \pi^k (\mathcal{G}_i), \boldsymbol{c}_i, \alpha_i \right\} \right), I_k \right),\tag{1}\] where \(\pi^k\) projects a splat into the \(k\)-th view, \(\mathcal{R}\) renders the projected splats by alpha-blending, and \(\mathcal{L}\) is the rendering loss function against the captured image \(I_k\). We seek to recover each splat by matching its projection to that of \(\mathcal{G}_i^*\) in every view for some distance metric \(d\) between 2D splats, \[\label{eq:gt-proj-opt} \left\{ \hat{\mathcal{G}}_i \right\} = \mathop{\mathrm{arg\,min}}_{\left\{ \mathcal{G}_i \right\}}\, \sum_{i \in [N]}\, \sum_{k \in [K]}\, d^2\!\left(\pi^k(\mathcal{G}_i),\, \pi^k(\mathcal{G}_i^*) \right).\tag{2}\]

Matching these projections is, under idealized conditions, sufficient to reconstruct the scene. As rendering depends on the splats only through the per-view inputs \(\{\pi^k(\mathcal{G}_i), \boldsymbol{c}_i, \alpha_i\}\) that \(\mathcal{R}\) consumes, any splats whose projections equal those of \(\mathcal{G}_i^*\) render exactly to the images that \(\mathcal{G}_i^*\) does, and thus also minimize 1 ³. That is, with proper per-view 2D targets, recovering the 3D splats reduces to the well-posed projection matching problem 2 , which we analytically solve in the backend (3.4).

In reality, however, the projections \(\pi^k(\mathcal{G}_i^*)\) are neither observable nor unique: there are infinitely many scenes that reproduce the captured images, and each can be partitioned into 2D splats in infinitely many ways. G\(^2\)SR therefore resolves this ill-posed subproblem by constructing one such set of 2D targets with its neural frontend by detecting them in the reference view (3.2) and tracking them to the remaining views (3.3).

3.2 Neural Splat Detection↩︎

We task a lightweight network \(\Phi\) with predicting, out of many configurations, a set of 2D Gaussian splats \(\{\bar{\mathcal{G}}_i^r\}\) for a chosen reference image \(I_r\) that, after triangulation, jointly represent the visible surfaces in \(I_r\). Like its 3D counterpart, each predicted 2D splat is an oriented Gaussian \(\bar{\mathcal{G}}_i^r = (\bar{\boldsymbol{\mu}}_i^r, \bar{\Sigma}_i^r)\) with center \(\bar{\boldsymbol{\mu}}_i^r \in \mathbb{R}^2\) and covariance \(\bar{\Sigma}_i^r = R(\theta_i^r)\,\bar{S}_i^r\,\bar{S}_i^{rT} R(\theta_i^r)^T\), factored into a 2D rotation \(R(\theta) \in \mathrm{SO}(2)\) and scales \(\bar{S} = \mathop{\mathrm{diag}}(\bar{s}_u, \bar{s}_v)\). During reconstruction, we take all but the last frame as the reference (2).

Since Gaussian splats are naturally supervised via the rendering loss, we make \(\Phi\) additionally predict photometric attributes \((\bar{\boldsymbol{c}}_i^r, \bar{\alpha}_i^r)\) per splat. Formally, \(\left\{ \bar{\mathcal{G}}_i^r, \bar{\boldsymbol{c}}_i^r, \bar{\alpha}_i^r \right\} = \Phi(I_r)\).

3.2.1 Architecture↩︎

For input image \(I_r\), \(\Phi\) first extracts a feature map of size \((H/s, W/s, 64)\) using a 3-layer convolutional network. A decoder head then iterates \(T\) times to predict \(T\) splats per \(s\times s\) block of \(I_r\), where each iteration involves: 1) generating one new splat token per block from the feature map; 2) concatenating with previously generated tokens; and 3) performing in-block cross-attention to resolve incompatibilities among splats in the same block. The tokens are mapped into a 10-dimensional vector encoding the splat parameters: \((\bar{\boldsymbol{\mu}}_i^r, \bar{s}_u, \bar{s}_v, \sin(\theta_i^r), \cos(\theta_i^r), \bar{\boldsymbol{c}}_i^r, \bar{\alpha}_i^r)\), finally resulting in a total of \(T \cdot (H/s) \cdot (W/s)\) splats for the whole image (2).

Unlike 3D attributes, such as depth or surface normal, that are predicted from information aggregated across multiple images, each splat predicted by \(\Phi\) inspects local texture and color and requires no capacity to memorize 3D scene structure or approximate multi-view geometry. This allows us to deploy a lightweight network with only 0.41 M parameters, which is \(6.2\times\) and \(53.7\times\) smaller than popular backbones such as MobileNetV3-Small and ViT-S, respectively, thus shrinking the memory footprint of neural inference ([sec:exp:efficiency:memory]).

3.2.2 Training↩︎

To ensure \(\{\bar{\mathcal{G}}_i^r\}\) can be triangulated into an accurate 3D representation of the visible surfaces, we desire \(\{\bar{\mathcal{G}}_i^r\}\) to: 1) cover most of the image, and 2) respect surface boundaries. Since surface boundaries usually correlate with those of color, we supervise \(\Phi\) end-to-end by penalizing differences in rasterized color with the ordinary \(\ell_1\) and structural similarity index measure (SSIM) losses from 3DGS [5]: \[\label{eq:detection-loss} \mathcal{L} = \lambda_1 \| \bar{I}_r - I_r \|_1 + \lambda_2 \left( 1 - {\mathrm{SSIM}} \left( \bar{I}_r,\; I_r \right) \right)\tag{3}\] where \(\bar{I}_r = \mathcal{R} \left( \left\{ \bar{\mathcal{G}}_i, \bar{\boldsymbol{c}}_i, \bar{\alpha}_i \right\} \right)\) is the image rasterized from predicted splats, and \(\lambda_1, \lambda_2\) are weights.

Under the rasterizer’s fixed front-to-back order, 3 does not effectively penalize boundary-violating splats at the end of the list due to their low visibility during alpha-blending. Such splats break the local-planarity assumption underlying our triangulation, yielding inconsistent targets under the warping of 3.3. We therefore randomly permute the predicted splats before rasterization, forcing \(\Phi\) to produce a set valid under any ordering and closing the caveat raised in 3.1. We train \(\Phi\) on the RealEstate10K dataset [28] due to its diverse distribution of image appearances.

3.3 Affine Splat Tracking↩︎

After detecting a set of 2D splats \(\{\bar{\mathcal{G}}_i^r\}\) that tile \(I_r\), we locate the respective projection targets in other views, i.e., 2D splats \(\{\bar{\mathcal{G}}_i^k\}_{k > r}\) that correspond to the same surfel in view \(k\).

As a change of viewpoint translates a splat’s center and deforms its contour due to perspective projection, we characterize such a cross-view correspondence by a local affine transform \((\boldsymbol{t}_i^k, A_i^k)\), with translation \(\boldsymbol{t}_i^k \in \mathbb{R}^2\) and linear mapping \(A_i^k \in \mathrm{GL}(2)\). Together, the two components map the support region \(\Omega(\bar{\mathcal{G}}_i^r)\) of the reference splat onto its counterpart \(\Omega(\bar{\mathcal{G}}_i^k)\) in view \(k\), where \(\Omega(\bar{\mathcal{G}}) = \{ \boldsymbol{x} : \norm{ \boldsymbol{x} - \bar{\boldsymbol{\mu}} }_{\bar{\Sigma}} \le 2 \}\) is the elliptical region within the 2-sigma radius. Once \((\boldsymbol{t}_i^k, A_i^k)\) is estimated, the corresponding 2D splat in view \(k\) is then assembled using first-order approximation as \[\bar{\mathcal{G}}_i^k := (\bar{\boldsymbol{\mu}}_i^k, \bar{\Sigma}_i^k) = (\bar{\boldsymbol{\mu}}_i^r + \boldsymbol{t}_i^k,\; A_i^k \bar{\Sigma}_i^r (A_i^k)^T).\]

To tackle this ill-posed correspondence subproblem, we estimate the affine parameters from the optical flow predicted by the efficient pre-trained network NeuFlowV2 [29]. Specifically, having obtained the per-pixel optical flow \(\boldsymbol{F}^{k \leftarrow r} \in \mathbb{R}^{H \times W \times 2}\), we define the translation to be the displacement of the splat center, \(\boldsymbol{t}_i^k = \boldsymbol{F}^{k \leftarrow r}(\bar{\boldsymbol{\mu}}_i^r)\) and estimate the linear mapping \(A_i^k\) using least-squares: Let \(\mathrm{Warp}(\boldsymbol{u}) = \boldsymbol{u} + \boldsymbol{F}^{k \leftarrow r}(\boldsymbol{u})\) denote the flow warping function. \(A_i^k\) is recovered as the best candidate that maps points in \(\Omega(\bar{\mathcal{G}}_i^r)\) to \(\mathrm{Warp}\left(\Omega(\bar{\mathcal{G}}_i^r)\right)\): \[A_i^k = \mathop{\mathrm{arg\,min}}_{A \in \mathrm{GL}(2)} \sum_{\boldsymbol{x} \in \Omega(\bar{\mathcal{G}}_i^r)} \norm{A \Delta\boldsymbol{x}^r - \Delta\boldsymbol{x}^k}_2^2, \label{eq:affine-correspondence}\tag{4}\] where \(\Delta\boldsymbol{x}^r = \boldsymbol{x} - \bar{\boldsymbol{\mu}}_i^r\) and \(\Delta\boldsymbol{x}^k = \mathrm{Warp}(\boldsymbol{x}) - \mathrm{Warp}(\bar{\boldsymbol{\mu}}_i^r)\) are the displacements of a support point \(\boldsymbol{x}\) from the center in view \(r\) and after warping to view \(k\), respectively.

Evaluating the objective in 4 over every pixel of \(\Omega(\bar{\mathcal{G}}_i^r)\) is both wasteful and sensitive to flow noise. Inspired by the unscented transform [30], we instead constrain test points \(\boldsymbol{x}\) of each splat to \(2n+1 = 5\) sigma points (for \(n = 2\) dimensions) placed at the center and along the principal axes: \[\boldsymbol{x}_0 = \bar{\boldsymbol{\mu}}; \;\;\; \boldsymbol{x}_{l = 1 \dots 4} = \bar{\boldsymbol{\mu}} + c_l L_l,\] where the spreads are \(c_1 = -c_2 = c_3 = -c_4 = \sqrt{2}\), \(L_1 = L_2\) is the first column of \(R(\theta) \bar{S}\), and \(L_3 = L_4\), the second, so that \(\boldsymbol{x}_{1\dots4}\) lie one standard deviation away on either side of each axis (2). As a linear least-squares problem, 4 under this approximation admits a closed-form solution by solving its \(4 \times 4\) normal equation. To reject unreliable correspondences, we additionally compute the reverse flow \(\boldsymbol{F}^{r \leftarrow k}\) to estimate the inverse translation \(\boldsymbol{t}_k^i\), and discard correspondences with \(\norm{\boldsymbol{t}_k^i - \boldsymbol{t}_i^k}_2 > 3\) pixels (forward–backward consistency). The splat tracking procedures are depicted in the middle column of 2.

3.4 Geometric Splat Triangulation↩︎

Given the corresponding 2D splats \(\{\bar{\mathcal{G}}_i^k\}_{k \in [K]}\) across all views, camera extrinsics \(\{T^k\}_{k \in [K]}\) and intrinsics \(C\), we recover each 3D splat \(\mathcal{G}_i\) independently by solving the well-posed projection matching problem \[\label{eq:g2sr-opt} \hat{\mathcal{G}}_i = \mathop{\mathrm{arg\,min}}_{\mathcal{G}_i}\, \sum_{k \in [K]}\, d_H^2\!\left(\pi^k(\mathcal{G}_i),\, \bar{\mathcal{G}}_i^k \right),\tag{5}\] instantiated from 2 by substituting the observed projection targets \(\bar{\mathcal{G}}_i^k\) for the unobservable ground-truth projections \(\pi^k(\mathcal{G}_i^*)\) and choosing the distance \(d\) to be the squared Hellinger distance between two Gaussians, \[d^2_H(\mathcal{G}_1, \mathcal{G}_2) = 1 - \frac{\det(\Sigma_1 \Sigma_2)^{1/4}}{\det(\Sigma_m)^{1/2}}\exp\!\left(-\frac{\norm{\boldsymbol{\mu}_1-\boldsymbol{\mu}_2}_{\Sigma_m}^2}{8}\right), \label{eq:hellinger}\tag{6}\] with \(\Sigma_m = \frac{1}{2}(\Sigma_1 + \Sigma_2)\). Smooth and bounded in \([0, 1]\), \(d_H^2\) jointly penalizes discrepancies in splat position, orientation, and scale within a single differentiable measure.

Since 5 admits no closed-form solution, we minimize it iteratively with the Gauss–Newton algorithm, with all splats optimized in parallel (subscript \(i\) dropped below for clarity). At each iteration, we evaluate the per-view Jacobian \[\require{physics} \boldsymbol{J}^k = \pdv{d^2_H}{\pi^k(\mathcal{G})} \circ \pdv{\pi^k(\mathcal{G})}{(\boldsymbol{\mu}, \boldsymbol{\Sigma})} \circ \pdv{(\boldsymbol{\mu}, \boldsymbol{\Sigma})}{(\boldsymbol{\mu}, R, s_{u}, s_{v})}\] and the residual \(r^k = d_H(\pi^k(\mathcal{G}),\, \bar{\mathcal{G}}^k)\), and solve for the update \(\delta \mathcal{G} = (\delta\boldsymbol{\mu}, \delta R, \delta s_{u}, \delta s_{v}) = -\boldsymbol{H}^{-1} \boldsymbol{b}\), where \[\boldsymbol{H} = \sum_{k \in [K]} (\boldsymbol{J}^k)^T \boldsymbol{J}^k,\qquad\boldsymbol{b} = \sum_{k \in [K]} (\boldsymbol{J}^k)^T r^k.\] The update is then applied to the respective parameters as \[\boldsymbol{\mu} \leftarrow \boldsymbol{\mu} + \delta \boldsymbol{\mu},\quad R \leftarrow R \operatorname{Exp}(\delta R),\quad s_{u,v} \leftarrow s_{u,v} e^{\delta s_{u,v}}\] where the exponential retraction \(\operatorname{Exp}\) updates \(R\) on the \(\mathrm{SO}(3)\) manifold and the multiplicative update keeps the scales positive. We further scale the update by a step size \(\eta\) to stabilize the optimization, and run up to 20 iterations.

Because \(d_H^2\) in 6 is nonlinear and nonconvex, the Gauss–Newton iterations require careful initialization of each parameter group to converge to the correct splat. We detail our initializations for mean, rotation, and scale below.

Mean. The exponential term in 6 saturates \(d_H\) to nearly 1 with a vanishing gradient when the projected and observed centers are far apart. We overcome this narrow convergence radius by triangulating \(\boldsymbol{\mu}\) using direct linear transformation (DLT) [31] triangulation from the observed 2D centers \(\{\bar{\boldsymbol{\mu}}^k\}\) and the camera parameters \(C\) and \(\{T^k\}\).

Rotation. Although corresponding splats \(\{\bar{\mathcal{G}}_i^k\}_{k \in [K]}\) from only 2 views already supply \(2 \times 5 = 10\) degrees-of-freedom (DoF) of geometric constraints, which is sufficient to determine the 8-DoF 3D splat, jointly recovering \(R\) and \((s_u, s_v)\) from \(d_H^2\) alone is ill-conditioned. For example, an elongated but slanted splat may produce a nearly identical projection \(\pi^k(\mathcal{G})\) as that of a round splat parallel to the image plane. This effect leaves the orientation only weakly observed under projection matching. We therefore exploit the relative affine map between view pairs, \(A^{kl} = A^k (A^l)^{-1}\), obtained from 4 , to initialize the splat’s normal direction \(R\boldsymbol{e}_z\) by \[R\boldsymbol{e}_z \leftarrow \mathop{\mathrm{arg\,min}}_{\norm{\boldsymbol{n}}_2 = 1} \norm{M \boldsymbol{n}}_2^2, \label{eq:triangulation:rot-init}\tag{7}\] where \(M \in \mathbb{R}^{6 \times 3}\) is constructed from \(A^{kl}\) and the Jacobians of the per-view projections \(\pi^k\) (Eq. (8) of [32]). Minimizing 7 seeks the normal \(\boldsymbol{n}\) for which the relative affine matrix \(\tilde{A}^{kl}(\boldsymbol{n})\) between view \(l\) and \(k\) induced by a surface orientation \(\boldsymbol{n}\) best matches the measured \(A^{kl}\) element-wise up to scale. The remaining DoF, i.e., the in-plane orientation of the \(\boldsymbol{e}_u\) axis about \(R\boldsymbol{e}_z\), is initialized randomly and refined in the subsequent Gauss–Newton iterations.

Scale. Due to the same orientation–scale ambiguity that makes splat extent \((s_u, s_v)\) difficult to recover from scratch, we initialize them by computing, for each view, the scaling that stretches \(\pi^k(\mathcal{G})\) to match the extents of \(\bar{\mathcal{G}}^k\) along its two axes, and take the minimum across views as a conservative estimate.

Because this initialization is derived purely from the observed projections, it adapts automatically to the metric scene scale and various camera configurations, allowing the pipeline to operate out of the box without the per-dataset fine-tuning commonly required by end-to-end methods.

4pt

4 Experiments↩︎

 

We test G\(^2\)SR against state-of-the-art (SOTA) end-to-end 3DGS methods: MVSplat [12], FreeSplat [13], MonoSplat [14], C3G [15], and SurfelSplat [16], on efficiency (memory and throughput) and geometric accuracy (depth and mesh error). Across 2- and 3-view reconstruction tasks on three datasets, G\(^2\)SR leads in geometric accuracy in almost all experiments while performing – reconstructions per second (RPS) on a single NVIDIA RTX 4090 GPU at \(384 \times 512\) resolution and only consumes – MB of GPU memory. Compared to baselines, G\(^2\)SR achieves a speedup of up to \(\times\) and memory reduction of –\(\times\). In the remainder of this section, [sec:exp:exp-setup] describes the experimental setup; [sec:exp:efficiency] quantifies the efficiency of each method, and [sec:exp:geom-acc] reports the accuracy. For completeness, we report view-synthesis metrics in [sec:exp:rend-acc], where G\(^2\)SR stays competitive in per-pixel and structural fidelity (PSNR, SSIM) while trailing in fine perceptual texture (LPIPS) (3).

4.1 Experiment Setup↩︎

 

4.1.1 Datasets↩︎

We benchmark on three datasets spanning complementary conditions. Replica [1] provides high-fidelity synthetic indoor renders with noiseless RGB and dense ground-truth depth for assessing the theoretical upper bound of reconstruction accuracy. ScanNet [2] captures real-world indoor scans with realistic image noise, lighting changes, and motion blur for probing robustness. DTU [17] provides object-centric captures with mesh ground truth for evaluating surface reconstruction quality under controlled lighting. For Replica and ScanNet, we adopt the splits processed by FreeSplat [13], which provide pre-sampled 2- and 3-view sets with rectified intrinsics and field of view. For DTU, we adopt the 2DGS [9] variant, which normalizes each scene to a unit bounding cube.

4.1.2 Tasks and Metrics↩︎

We evaluate two reconstruction tasks: depth estimation on Replica and ScanNet, and mesh reconstruction on DTU. In depth estimation, each method is given a set of 2 or 3 views designated by FreeSplat [13]’s evaluation protocol as context to predict the Gaussian splats representing the scene; the splats are then rendered into depth maps at another set of target views and compared against the ground truth depth maps. We report Absolute Difference (Abs Diff, m), Absolute Relative Error (Abs Rel, %), and the inlier ratio within \(\pm 25\%\) of ground truth (\(\delta_1\), %) on target views. For mesh reconstruction, we evaluate all 15 DTU scans and report the mean Chamfer Distance (CD, mm) between the ground-truth and reconstructed meshes. We construct the mesh in two ways: TSDF fusion on rendered depth, following NeuS [33] with context views \(\{23, 24, 33\}\); and Poisson reconstruction on splat centers, following SurfelSplat [16] with context views \(\{0, 2, 9\}\). While reporting both for completeness, we note that the more widely adopted TSDF protocol renders intermediate depth maps from the splats and therefore holistically tests scale and rotation with the center, whereas Poisson isolates the accuracy of splat centers. We adopt a resolution of \(384 \times 512\) for depth estimation and \(480 \times 640\) for mesh reconstruction.

In addition to accuracy metrics, we report two quantities to characterize the efficiency: Memory (MB) is the peak size of GPU memory allocated during a single 2- or 3-view reconstruction, comprising model weights, activations, and other intermediate buffers, as recorded via PyTorch’s memory allocation trace. Reconstructions per second (RPS) measures the rate of the same. For DTU mesh reconstruction, meshing time is excluded so that RPS reflects splat inference alone.

We additionally report view-synthesis metrics, including PSNR, SSIM, and LPIPS for completeness, although G\(^2\)SR is not optimized to produce color-accurate splats that capture fine-grained textural details as baselines do. All methods are evaluated over valid pixels with rendered opacity \(\alpha \geq 0.5\) and report the resulting coverage (the percentage of such pixels), so that unreconstructed regions are not counted against a method.

4.1.3 Baselines↩︎

We compare against five SOTA end-to-end few-view reconstruction methods spanning the common backbone choices: MVSplat [12] (Custom CNN + Swin attention, hybrid), FreeSplat [13] (EfficientNet, CNN), MonoSplat [14] (DepthAnythingV2-S, ViT), C3G [15] (VGGT-1B, ViT), and SurfelSplat [16] (DINOv2-L, ViT). All baselines are evaluated using their released checkpoints and default hyperparameters.

4.1.4 Implementation Details↩︎

  We only train the splat detection network \(\Phi\) (3.2) on RealEstate10K [28] and use the pretrained NeuFlowV2 [29] optical flow network off-the-shelf. No fine-tuning is performed on Replica, ScanNet, or DTU. The Gauss-Newton solver runs for \(N=20\) iterations.

When evaluating depth estimation, since the baselines do not always predict splats in metric scale, we median-align their predictions to the ground truth before computing depth metrics to make a meaningful comparison⁴. In contrast, G\(^2\)SR produces metric-scale depth by construction through triangulation and is evaluated unaligned. Every depth result reported below thus reflects a fairness advantage granted to the baselines. All evaluations are conducted on a desktop workstation with a single NVIDIA RTX 4090 GPU and a 24-core Intel i9-14900K CPU. This platform serves as a controlled measurement bench on which every method, including those requiring high memory and compute, runs unconstrained. We expect the accuracy, memory footprint, and relative throughput to transfer across platforms and defer mobile benchmarking to future work.

4.2 Memory and Throughput↩︎

 

4.2.1 Memory↩︎

  [tab:depth-est-replica-scannet,tab:dtu-mesh] report the peak GPU memory allocated during inference, comprising model weights, activations, and intermediate buffers. G\(^2\)SR’s frontend, consisting of the 0.41 M-parameter detector and NeuFlowV2, accounts for 27 MB of weights; activations peak at \(\sim\) MB on 2-view inputs and \(\sim\) MB on 3-view inputs, for a total of – MB per reconstruction under an image resolution of \(384 \times 512\). On the same workloads, the baselines consume a memory of  MB to  GB, corresponding to a memory reduction of –\(\times\) across the experiments. By offloading well-posed structural inference to the geometric backend, our frontend is significantly smaller than the baselines’, whose weights alone consume  MB (MonoSplat) –  GB (C3G).

4.2.2 Throughput↩︎

  G\(^2\)SR runs at – RPS on depth estimation, with the lower end corresponding to 3-view inputs, and at  RPS on DTU mesh reconstruction. In comparison, the baselines achieve – RPS and – RPS on the two tasks, respectively. The –\(\times\) speedup arises from the same architectural shift that yields the memory savings: the geometric backend both allows for a simpler neural frontend and performs structural inference analytically that may require multiple neural layers to approximate.

4.3 Geometric Accuracy↩︎

 

4.3.1 Depth Estimation↩︎

  ¿tbl:tab:depth-est-replica-scannet? reports depth accuracy on Replica and ScanNet under 2- and 3-view inputs. On Replica 2-view, G\(^2\)SR attains an absolute difference of  m, relative error of %, and \(\delta_1\) of %, which roughly halves the error of the strongest baseline (FreeSplat) and retains a similar margin across the other three settings. Notably, G\(^2\)SR does so without the median alignment granted to the baselines ([sec:exp:exp-setup:impl-details]), confirming that it produces metric-scale depth consistent across views. 3 shows a representative Replica 2-view reconstruction: aside from boundary pixels lacking covisibility in context views and minor gaps between splats, G\(^2\)SR recovers most of the scene at uniformly low error, whereas end-to-end methods struggle on complex texture (e.g., the art painting on the wall) and at large depths (e.g., the back of the room). These failure patterns reflect where the depth comes from: a network that regresses depth from learned priors is biased toward its training distribution, compressing unfamiliar depth ranges and letting image texture bleed into geometry, whereas G\(^2\)SR solves depth from the projective constraints, so its residual error stems from 2D correspondence noise rather than from a learned 3D bias.

4.3.2 Mesh Reconstruction↩︎

  Under the widely adopted NeuS protocol (TSDF fusion), G\(^2\)SR achieves the lowest Chamfer distance of  mm, ahead of FreeSplat (), MonoSplat (), and the rest; under the SurfelSplat protocol (Poisson reconstruction on splat centers), G\(^2\)SR ( mm) is second only to SurfelSplat ( mm). We note that TSDF is the more demanding protocol, as it renders intermediate depth from the splats and thus tests scale and rotation alongside the center, requiring all splat parameters to be jointly consistent. Most baselines degrade noticeably from Poisson to TSDF, whereas G\(^2\)SR stays nearly unchanged, indicating that G\(^2\)SR recovers the full splat geometry, instead of centers only.

4.3.3 Robustness to Camera Geometry↩︎

  Solving the well-posed 3D triangulation analytically also makes G\(^2\)SR robust to shifts in camera geometry without diversified training or finetuning. The baselines fail under two shifts: (i) Intrinsics—methods trained on low-focal-length RealEstate10K-style data degrade (MVSplat, MonoSplat) or yield no meaningful CD (C3G) on DTU’s \(\sim2.5\times\) longer focal length; SurfelSplat, conversely, is trained on DTU but fails on indoor scenes ([tab:depth-est-replica-scannet,fig:visualizations]). (ii) View configuration—even within DTU, SurfelSplat degrades by \(\sim5\times\) across protocols ( to  mm) as the test views \(\{23, 24, 33\}\) differ from its training views \(\{0, 2, 9\}\). In comparison, trained only on RealEstate10K with NeuFlowV2 off-the-shelf, G\(^2\)SR is unaffected in both cases as the backend resolves camera parameters automatically and leaves the frontend focused on camera-agnostic 2D tasks.

4pt

4.4 Rendering Accuracy↩︎

 

For completeness, ¿tbl:tab:rend-acc? reports PSNR, SSIM, and LPIPS for 3-view reconstruction on Replica and ScanNet, evaluated over each method’s valid pixels. G\(^2\)SR reaches a PSNR of (Replica) and (ScanNet), which is – dB behind the SOTA FreeSplat but still within \(\sim\) dB of the popular MVSplat, confirming that it recovers appearances at a high level. It trails the leading baselines on fine texture (LPIPS) but still outperforms C3G and SurfelSplat. This gap follows from the different supervision target: while baselines learn to minimize rendering loss directly against ground-truth pixels, our formulation in 5 fits splats to detected 2D targets that themselves only approximate the images and thus does not guarantee the transfer of fine-grained texture.

G\(^2\)SR’s lower coverage (% / % versus above 94% for most baselines) stems from the aggressive outlier rejection, which discards splats with inconsistent forward–backward or out-of-bounds warps (3.3). While leaving boundary and occluded regions unreconstructed (grey in 3) that an end-to-end method would inpaint, it safeguards geometric accuracy by avoiding hallucinated structure; the missing coverage can be recovered by accumulating over more views.

5 Conclusion↩︎

We presented G\(^2\)SR, an efficient framework for Gaussian-based few-view surface reconstruction that decouples the pipeline into a lightweight neural frontend for detecting and tracking 2D Gaussian splats across views, and an analytic geometric backend that triangulates them into metric-scale 3D splats. By reserving learning for the ill-posed image-plane tasks and solving the well-posed 3D structure analytically, G\(^2\)SR keeps its networks small and focused on camera-agnostic 2D reasoning. On Replica, ScanNet, and DTU, G\(^2\)SR matches or surpasses the depth estimation and mesh reconstruction accuracy of state-of-the-art end-to-end methods while running at – RPS with only – MB of GPU memory on an RTX 4090 GPU (2- and 3-view, resolution \(> 384 \times 512\)), which is up to \(\times\) less than prior methods and makes it a practical step toward online, high-fidelity Gaussian-based surface reconstruction on resource-constrained platforms.

References↩︎

---

1. The authors are with the Massachusetts Institute of Technology, Cambridge, MA 02139, USA (e-mail: dasongg@mit.edu; sze@mit.edu; sertac@mit.edu).↩︎

2. We ignore view-dependent spherical-harmonic color for simplicity.↩︎

3. Up to rendering order; 3.2 alleviates this constraint by training the detector to predict 2D splats that render correctly under any ordering.↩︎

4. C3G internally normalizes the camera baseline to unit length, while the others inherit the depth scale from their training data. However, note that the ground truth is not available during online reconstruction.↩︎