1 Introduction↩︎

Recent advancements in 3D reconstruction and novel view synthesis (NVS) have been driven by Neural Radiance Fields (NeRFs) [1] and 3D Gaussian splatting (3DGS)  [2]. A standard training pipeline for novel view synthesis reconstructs a 3D scene from input views and optimizes it by aligning rendered novel views with ground-truth images [3]–[8].

State-of-the-art methods typically employ geometry-aware architectures by constructing cost volumes [3], [4], [6], leveraging epipolar transformers [5], or encoding camera poses using Plücker ray embeddings [9]–[11]. These approaches rely on camera poses estimated with Structure-from-Motion (SfM)[12] to reconstruct 3D scenes, as illustrated in Fig. 1 (a). However, acquiring camera poses from SfM is computationally expensive and often unreliable in sparse-view scenarios due to insufficient correspondences, limiting the applicability of these pose-required methods. To address this, recent research has focused on novel view synthesis under pose-free settings.

Existing pose-free methods reconstruct 3D scenes from unposed images by learning in a canonical space [7], [8], [13], [14], leveraging latent scene representations [15], [16], or jointly optimizing both context-view camera poses and 3D scene representations [17]–[19]. Although these methods do not require accurate poses at inference, their training is still supervised by rendering losses given ground-truth poses at novel viewpoints, as shown in Fig. 1 (b). We therefore categorize these approaches as supervised pose-free methods. As a result, their reliance on training datasets with known camera poses restricts the scalability to large-scale real-world data without pose annotations.

This raises a critical question: Are ground-truth poses truly indispensable for optimizing 3D scene representations during training? One solution is to use estimated poses at novel viewpoints, referred to as the self-supervised pose-free paradigm in Fig. 1 (c). However, this presents an inherent challenge: since the rendering loss intrinsically couples the learning of 3D scene geometry and camera poses, pose errors can degrade reconstruction quality, which further hampers pose estimation. Such mutual dependency creates a feedback loop that can potentially lead to unstable training or even divergence. Recent self-supervised pose-free approaches [20], [21] struggle to mitigate this issue primarily due to their use of separate and cascading modules for scene reconstruction and pose estimation, discouraging the learning of consistent feature representations across the two tasks and impairing geometric consistency. Consequently, these methods exhibit poor training stability, particularly under large viewpoint changes, and still lag far behind state-of-the-art pose-required and supervised pose-free methods [5]–[7].

To address the challenge, we introduce SPFSplatV2, a self-supervised pose-free approach for 3D Gaussian splatting from unposed sparse views. As shown in Fig. 2, SPFSplatV2 employs a shared backbone for feature extraction with dedicated heads for predicting 3D Gaussian primitives and camera poses relative to a reference view. The unified backbone improves computational efficiency and facilitates joint feature learning for scene reconstruction and pose estimation, thereby enhancing geometric consistency and mitigating feedback instability. This is achieved by enabling 3D geometry to benefit from context-aware camera alignment and allowing pose predictions to leverage global scene context.

During training, in addition to context images, the target images are also incorporated as input for target pose estimation, enabling rendering losses at target views. To prevent information leakage from target images into the Gaussian reconstruction of context views, we introduce a masked attention mechanism, as shown in Fig. 2. In this design, context tokens attend only to context tokens, ensuring that 3D Gaussian reconstruction remains independent of target-view information. Conversely, target tokens attend to both context and target tokens, allowing the model to exploit global scene context for accurate target pose estimation. Finally, we complement the rendering loss with a reprojection loss that explicitly enforces alignment between the predicted Gaussians and their corresponding image pixels, imposing stronger geometric constraints and further enhancing training stability. In conclusion, we make the following key contributions:

• We propose SPFSplatV2, a feed-forward framework with masked attention that enables efficient and stable joint optimization of scene reconstruction and pose estimation from sparse unposed views, requiring no ground-truth poses during training and inference.

• SPFSplatV2 outperforms state-of-the-art pose-required, supervised pose-free, and self-supervised pose-free methods on both in-domain and out-of-domain novel view synthesis, demonstrating robustness under limited view overlap and extreme viewpoint changes. Despite relying solely on image supervision, its efficient feed-forward relative pose estimation surpasses many approaches that depend on geometric supervision.

• By eliminating the reliance on ground-truth poses during training, our method offers the scalability needed to leverage larger and more diverse datasets. Its effectiveness across different architectures further demonstrates the paradigm’s broad compatibility.

This work substantially extends our previous method, SPFSplat [22], with the key novelties summarized as follows:

• Methodological Improvements: Different from SPFSplat, which employs separate context-only and context-with-target input branches to avoid target information leakage, we introduce a unified architecture with masked attention mechanism that reduces computational overhead and pose misalignment. We further adopt learnable pose tokens for pose estimation to ensure compatibility with the VGGT architecture. In addition, a multi-view dropout strategy enhances generalization across varying numbers and spatial distributions of context views.

• Architectural Compatibility: We demonstrate that our training paradigm is compatible with state-of-the-art reconstruction models. To this end, we develop two variants: SPFSplatV2, which follows a MASt3R-style [23] architecture (consistent with SPFSplat), and SPFSplatV2-L, which adopts the VGGT [24] architecture.

• Superior Performance: Extensive experiments show that SPFSplatV2 and SPFSplatV2-L achieve significant improvements over SPFSplat [22] and other state-of-the-art methods in novel view synthesis, cross-domain generalization and relative pose estimation.

2 Related Work↩︎

2.1 Novel View Synthesis↩︎

NeRF [1] and 3DGS [2] have demonstrated strong performance in 3D reconstruction and novel view synthesis (NVS). Early methods rely on dense input views for per-scene optimization [25]–[28], whereas recent approaches enable generalizable reconstruction from sparse-view images [3]–[11], [29]–[31]. Typical NVS pipelines reconstruct 3D scenes from input views and optimize them by aligning synthesized images to ground-truth targets. Based on their dependence on ground-truth camera poses during training and inference, existing methods can be grouped into pose-required, supervised pose-free, and self-supervised pose-free approaches, as in Fig. 1.

Pose-Required Methods reconstruct 3D scenes from images given accurate poses using geometry-aware architectures [3]–[6], [9]–[11], [30]. For example, MVSNeRF [3] and MuRF [4] construct cost volumes for multi-view aggregation to reconstruct radiance fields, while MVSplat [6] uses cost volumes for depth estimation to reconstruct Gaussian primitives. Other strategies include epipolar transformers in pixelSplat [5] or encoding camera poses with Plücker ray embeddings [9]–[11] or leveraging pretrained depth priors [30]. Despite their effectiveness, these methods depend on SfM for precise camera poses, which is computationally expensive and often unreliable in sparse-view scenarios. Recent pose estimation techniques [23], [32]–[35] mitigate some issues but still struggle in low-overlap or texture-less settings. Consequently, pose-required methods remain impractical for unposed reconstruction during both training and inference.

Supervised Pose-Free Methods enable 3D reconstruction from unposed images, relaxing the need for camera poses at inference. Methods such as UpSRT [16] and UpFusion [15] encode unposed images into latent scene representations, while BARF [19], SPARF [36], DBARF [18], and CoPoNeRF [17] jointly optimize poses and NeRF representations. LEAP [13] and PF-LRM [14] leverage ViT architectures to define neural volumes in canonical camera coordinates. More recently, Splatt3R [8] predicts 3D Gaussians in a canonical space by regressing offsets to pointmaps from a frozen MASt3R [23], but requires depth supervision. NoPoSplat [7] removes this depth reliance and refines this pipeline by fine-tuning MASt3R and incorporating intrinsics to mitigate scale ambiguity. YoNoSplat [29] provides a flexible architecture that supports both pose-required and pose-free input settings. However, despite removing pose requirements at inference, these methods still depend on ground-truth poses during training via rendering losses [7], [8], [13]–[16], [29], [30], explicit pose supervision [17], [29], or coarse pose initialization [19], [36], therefore limiting scalability to large-scale unposed data.

Self-Supervised Pose-Free Methods completely eliminate the reliance on ground-truth poses during training by enabling rendering losses at novel viewpoints using estimated poses, as shown in Fig. 1 (c). For instance, Nope-NeRF [37], CF-3DGS [38], and FlowCam [39] reconstruct 3D scenes and estimate camera poses incrementally by re-rendering dense video sequences. However, they are limited to continuous video frames and do not generalize well to sparse views. Recent self-supervised pose-free methods, such as PF3plat [20] and SelfSplat [21], attempt to estimate both input- and target-view poses from sparse views. PF3plat relies on off-the-shelf feature descriptors [40] together with RANSAC-based initialization, resulting in a pipeline that is computationally expensive and not end-to-end trainable. In contrast, SelfSplat employs cross-view U-Nets [41], [42] for pose prediction, but its performance degrades substantially under large viewpoint changes. Beyond the limitations of pose estimation, both methods decouple pose prediction and Gaussian reconstruction into separate modules, preventing feature sharing between the two tasks and leading to weaker geometric alignment and higher computational overhead. AnySplat [43] also jointly optimizes 3D Gaussians and camera poses. However, its rendering loss is applied only to the input context views, providing no direct supervision for novel viewpoints during training. As a result, novel view synthesis performance remains limited, and the method relies on continuous distillation from VGGT throughout training to stabilize depth and camera pose estimation.

SPFSplat [22], our previous approach, also adopts a self-supervised, pose-free paradigm by jointly optimizing 3D Gaussians and camera poses through a shared backbone in a canonical space. Guided by image rendering and reprojection losses, the shared representation promotes geometric consistency and stable optimization. Unlike AnySplat, SPFSplat uses self-predicted target poses to render and supervise target-view observations, providing direct supervision for novel viewpoints. Building upon SPFSplat, we introduce masked attention and additional architectural refinements, resulting in a unified self-supervised framework that is compatible with different backbone designs while improving performance and generalization across varying numbers of input views.

2.2 Structure-from-Motion (SfM)↩︎

Structure-from-Motion (SfM)[44], [45] is a core problem in computer vision that jointly estimates camera parameters and reconstructs sparse 3D maps from image collections. Classical SfM pipelines typically involve local feature detection and matching[46]–[48], geometric verification via epipolar geometry or homographies with RANSAC [49], triangulation [50] to recover 3D points, and bundle adjustment [51] to refine poses and structure. Recent advances have incorporated learning-based components into SfM, including robust feature descriptors [52]–[55], improved image matching [40], [56], detector-free matching [57], and neural bundle adjustment [19], [58]. However, the sequential design of SfM pipelines remains prone to error propagation. To overcome this, fully differentiable pipelines have been introduced [23], [24], [32], [59]–[61]. For example, VGGSfM [59] enables end-to-end sparse reconstruction, while DUSt3R [32], following the architecture of CroCo [62], performs dense 3D reconstruction without camera parameters. MASt3R [23] further enhances feature matching and local representations but, like DUSt3R, remains constrained by pairwise architectures and costly global optimization, limiting its applicability to efficient multi-view reconstruction. Extensions such as MV-DUSt3R+[60] and Fast3R[61] address multi-view reconstruction, while FLARE [63] employs cascaded learning with pose as the central bridge. Recently, VGGT [24] introduces a feed-forward transformer that jointly infers camera parameters, depth, point maps, and 3D tracks, achieving state-of-the-art results.

Similar to these SfM methods, our method jointly predicts 3D points and poses, with rendering and reprojection losses acting as a differentiable form of bundle adjustment to refine both geometry and poses. Unlike prior work, it requires no ground-truth geometric priors during training. In addition, the training paradigm is naturally compatible with advanced reconstruction backbones such as MASt3R [23] and VGGT [24], giving rise to two variants: SPFSplatV2 and SPFSplatV2-L.

3 Method↩︎

We aim to learn a feed-forward network that reconstructs 3D Gaussians from unposed images while simultaneously estimating camera poses. During training, the predicted poses are used to render target views for photometric supervision, enabling joint optimization of geometry and camera poses without requiring ground-truth pose annotations.

3.1 Problem Formulation↩︎

Consider \(N\) context images \(\{\boldsymbol{I}^v\}_{v=1}^{N}\) as input. During training, additional \(M\) target images \(\{\boldsymbol{I}^v\}_{v=N{+}1}^{N{+}M}\) are provided, resulting in a total of \(V = N{+}M\) views.

3D Gaussian Reconstruction: Following [7], [8], we predict 3D Gaussians from context images in a canonical 3D space where the first input view \(\boldsymbol{I}^1\) serves as the global coordinate frame. The reconstruction network is formulated as: \[f_{\boldsymbol{\theta}} : \{\boldsymbol{I}^v\}_{v=1}^{N} \mapsto \{ \boldsymbol{\mathcal{G}}^{v \rightarrow 1}\}_{v=1}^{N}, \label{eq:gaussian95formulation}\tag{1}\] where \(\boldsymbol{\mathcal{G}}^{v \rightarrow 1} = \{(\boldsymbol{\mu}_j^{v \rightarrow 1}, \boldsymbol{r}_j^{v \rightarrow 1}, \boldsymbol{s}_j^v, \boldsymbol{c}_j^{v \rightarrow 1}, \alpha_j^v)\}_{\substack{j=1,\dots,H \times W}}\) represents the pixel-aligned Gaussians for \(\boldsymbol{I}^v\), represented in the coordinate frame of \(\boldsymbol{I}^1\). Each Gaussian is parameterized by center \(\boldsymbol{\mu} \in \mathbb{R}^3\), rotation quaternion \(\boldsymbol{r} \in \mathbb{R}^4\), scale \(\boldsymbol{s} \in \mathbb{R}^3\), opacity \(\alpha \in \mathbb{R}\), and spherical harmonics (SH) \(\boldsymbol{c} \in \mathbb{R}^k\) with \(k\) degrees of freedom.

Pose Estimation: We introduce a pose network \(f_{\boldsymbol{\phi}}\) to estimate the relative transformation from each view \(\boldsymbol{I}^v\) to the reference view \(\boldsymbol{I}^1\), which is denoted as \(\boldsymbol{P}^{v \to 1} =[\boldsymbol{R}^{v \to 1} | \boldsymbol{T}^{v \to 1} ]\), where \(\boldsymbol{R}^{v \to 1} \in \mathbb{R}^{3 \times 3}\) represents the rotation matrix, and \(\boldsymbol{T}^{v \to 1} \in \mathbb{R}^{3 \times 1}\) represents the translation vector. This can be formulated as: \[\boldsymbol{P}^{v \to 1} = f_{\boldsymbol{\phi}}(\boldsymbol{I}^{v},\dots, \boldsymbol{I}^{1}), v \in [1, \dots, V]. \label{eq:pose95head}\tag{2}\]

Novel View Synthesis: Novel views are then rendered using the estimated target poses and reconstructed Gaussians: \[\begin{align} \hat{\boldsymbol{I}^t} &= \mathcal{R}(\boldsymbol{P}^{t \to 1}, \{\boldsymbol{\mathcal{G}}^{v \rightarrow 1}\}_{v=1}^{N}), & t \in [N{+}1, \dots, V]. \end{align} \label{eq:image95rendering}\tag{3}\]

3.2 Architecture↩︎

Following state-of-the-art large reconstruction models such as MASt3R [23] and VGGT [24], our framework consists of three main components: an encoder, a decoder, and task-specific prediction heads, as illustrated in Fig. 2. Both the encoder and decoder follow Vision Transformer (ViT) architectures [64]. As shown in Fig. 3, we develop two model variants: SPFSplatV2, which adopts the MASt3R-style architecture, and SPFSplatV2-L, which follows the VGGT design. In the following, we introduce both variants in detail.

Encoder: The RGB image \(\boldsymbol{I}^v\) for each input view \(v\) is first patchified and flattened into a sequence of image tokens. These tokens are independently processed by a shared-weight ViT encoder, which extracts view-specific feature representations \(\boldsymbol{F}^{v} \in \mathbb{R}^{L \times C}\), where \(L\) denotes the number of tokens. This is formulated as follows: \[\boldsymbol{F}^{v} = \mathrm{Encoder}(\boldsymbol{I}^{v}), v \in [1, \dots, V]. \label{eq:encoder}\tag{4}\]

Learnable Pose Token: MASt3R does not provide a dedicated pose estimation head, and the original SPFSplat [22] estimates camera poses through global average pooling of the decoded image tokens. To maintain compatibility with VGGT and provide a unified pose estimation design across architectures, we adopt learnable pose tokens in both SPFSplatV2 and SPFSplatV2-L. SPFSplatV2 introduces a learnable pose token \(\boldsymbol{g} \in \mathbb{R}^{1 \times C}\), which is replicated for each view \(v \in [1,V]\) as \(\boldsymbol{g}^v\). Unlike SPFSplatV2 which uses an asymmetrical decoder to distinguish the reference frame from the other views, SPFSplatV2-L introduces two separate learnable pose tokens \(\bar{\boldsymbol{g}}\) and \(\bar{\bar{\boldsymbol{g}}} \in \mathbb{R}^{1 \times C}\), following VGGT. Specifically, \(\bar{\boldsymbol{g}}\) is assigned to the reference frame (\(\boldsymbol{g}^1 := \bar{\boldsymbol{g}}\)), while all other views share \(\bar{\bar{\boldsymbol{g}}}\) (\(\boldsymbol{g}^v := \bar{\bar{\boldsymbol{g}}}, v \in [2, \dots, V]\)). The pose tokens are concatenated with the encoder tokens, yielding \(\boldsymbol{F}^{v} := [\boldsymbol{g}^v, \boldsymbol{F}^{v}]\). In the decoding stage, the learnable pose tokens can selectively attend to the informative features.

Intrinsics Embedding: Following [7], we encode the camera intrinsics of each view into a token \(\boldsymbol{k}^v\) via a linear layer and concatenate it with the pose token and encoder tokens, forming the decoder input \(\boldsymbol{F}^v := [\boldsymbol{k}^v, \boldsymbol{g}^v, \boldsymbol{F}^v]\). This explicitly injects calibration information, helping to resolve scale ambiguity and improve alignment of predicted poses and 3D Gaussians, particularly under large focal length variations. Importantly, the intrinsic token is optional, and SPFSplatV2 maintains strong performance without it (Sec. 4.6), underscoring the robustness and flexibility of the design.

Masked Multi-view Decoder: To effectively aggregate information across multiple views, we adopt a ViT-based decoder with cross-view attention, enabling joint reasoning over token representations across input views and facilitating cross-view information exchange to capture spatial relationships and the global 3D scene geometry.

During training, since target views are also provided as input, the original SPFSplat [22] adopts a dual-branch design to avoid information leakage from target views into the Gaussian reconstruction of context views. One branch processes only context images for Gaussian prediction, while the other takes both context and target images for pose estimation. However, this design introduces two drawbacks: (i) higher computational cost in cross-attention caused by two forward passes during training, as shown in Fig. 4 (a), and (ii) target poses and 3D geometry are predicted in separate branches, which may lead to pose-geometry misalignment.

To address these issues, we introduce a masked attention mechanism as shown in Fig. 3 and Fig. 4 (b). In this approach, context and target images are jointly processed in a single forward pass, with cross-attention selectively masked to control information flow. Specifically, context tokens attend only to context tokens, ensuring Gaussian reconstruction remains independent of target-view information. Meanwhile, target tokens attend to both context and target tokens, allowing the network to leverage global cues for accurate pose estimation.

For SPFSplatV2, we extend MASt3R’s pairwise asymmetric decoder to a multi-view setting, which scales efficiently with the number of views while avoiding excessive memory overhead, following similar implementations in [7], [60]. Each decoder block first performs intra-view self-attention, followed by masked cross-attention. Tokens from the first (reference) view are processed with \(\mathrm{DecoderBlock}^1\), while tokens from the remaining views use \(\mathrm{DecoderBlock}^2\). The two decoders share the same architecture but maintain independent weights. Formally, the masked decoder block is defined as: \[\small \boldsymbol{G}^{v}_{i} = \begin{cases} \mathrm{DecoderBlock}_i^1(\boldsymbol{G}^{v}_{i-1}, \boldsymbol{G}^{1:K}_{i-1}), & v=1, \\[3pt] \mathrm{DecoderBlock}_i^2(\boldsymbol{G}^{v}_{i-1}, \boldsymbol{G}^{1:K}_{i-1}), & v \in [2, \dots, V], \\ \end{cases} \label{eq:v295decoder}\tag{5}\] for \(i = 1, \dots, B\), where \(B\) is the number of decoder blocks and \(\boldsymbol{G}^{v}_{0}=\boldsymbol{F}^v\) are the initial tokens for view \(v\). For context views (\(v \in [1,\dots,N]\)), we set \(K=N\) such that attention is restricted to the context views. For target views (\(v \in [N+1,\dots,V]\)), we set \(K=V\), allowing attention over all views.

For SPFSplatV2-L, we adopt the VGGT architecture, which alternates between intra-frame and inter-frame attention. Different from MASt3R’s asymmetric design, the decoder here is unified across all views, which can be expressed as:

\[\small \boldsymbol{G}^{v}_{i} = \mathrm{DecoderBlock}_i\left(\boldsymbol{G}^{v}_{i-1}, \boldsymbol{G}^{1:K}_{i-1}\right), \quad v \in [1, \dots, V], \label{eq:v2l95decoder}\tag{6}\] for \(i = 1, \dots, B\), where \(B\) is the number of decoder blocks, and \(K\) follows the same definition as in SPFSplatV2.

Overall, the masked multi-view decoder achieves three key advantages: (i) it preserves generalization to novel viewpoints by strictly preventing target-specific information from contaminating the Gaussian representation, (ii) it significantly reduces computational overhead by avoiding redundant forward passes, and (iii) it predicts 3D geometry and target poses from a unified decoder representation, thereby improving pose-geometry alignment and reducing pose drift. Together, these improvements lead to more efficient and stable training, as well as more accurate reconstructions.

Gaussian Prediction Heads: Following [7], [8], we employ two DPT-based heads [65] to infer Gaussian parameters. The first head processes decoder tokens of context views and predicts 3D coordinates for each pixel, defining Gaussian centers. The second head estimates rotation, scale, opacity, and SH coefficients for each Gaussian primitive.

For SPFSplatV2, the Gaussian center head extends MASt3R’s pairwise asymmetric pointmap head to a multi-view setting by assigning decoder tokens from the first view to the reference head \(\mathrm{PointHead}^1\), and tokens from all remaining views to the non-reference head \(\mathrm{PointHead}^2\). The Gaussian parameter head follows the same structure as the Gaussian center head. As proposed in [5]–[7], we incorporate high-resolution skip connections by feeding the original context images into the Gaussian parameter heads, preserving fine-grained spatial details. These heads can be formulated as:

\[\begin{align} \boldsymbol{\mu}^{ v \rightarrow 1} &= \begin{cases} \mathrm{PointHead}^1\!\left(\{\boldsymbol{G}^{v}_{i}\}_{i=0}^{B}\right), & v = 1, \\ \mathrm{PointHead}^2\!\left(\{\boldsymbol{G}^{v}_{i}\}_{i=0}^{B}\right), & v \in [2, \dots, N], \end{cases} \tag{7} \\[10pt] \boldsymbol{\overline{\mathcal{G}}}^{v \rightarrow 1} &= \begin{cases} \mathrm{GSHead}^1\!\left(\{\boldsymbol{G}^{v}_{i}\}_{i=0}^{B}, \boldsymbol{I}^{v}\right), & v = 1, \\ \mathrm{GSHead}^2\!\left(\{\boldsymbol{G}^{v}_{i}\}_{i=0}^{B}, \boldsymbol{I}^{v}\right), & v \in [2, \dots, N], \end{cases} \tag{8} \end{align}\] where \(\{\boldsymbol{G}^{v}_{i}\}_{i=0}^{B}\) denotes the set of decoder tokens taken from different blocks, \(\boldsymbol{\mu}^{ v \rightarrow 1}\) denotes Gaussian centers, and \(\boldsymbol{\overline{\mathcal{G}}}^{v \rightarrow 1}= \{(\boldsymbol{r}_j^{v \rightarrow 1}, \boldsymbol{c}_j^{v \rightarrow 1}, \alpha_j^v, \boldsymbol{s}_j^v)\}\) represents rotation, scale, opacity, and SH coefficients for each Gaussian primitive.

For SPFSplatV2-L, we adopt the VGGT design for the pointmap head, which serves as both the Gaussian center head and the Gaussian parameter head. Similar to SPFSplatV2, the Gaussian parameter head also incorporates the original context images as an auxiliary input. Unlike the asymmetric design in SPFSplatV2, the Gaussian prediction heads in SPFSplatV2-L are unified across all views: \[\begin{align} \boldsymbol{\mu}^{ v \rightarrow 1} &= \mathrm{PointHead}\!\left(\{\boldsymbol{G}^{v}_{i}\}_{i=0}^{B}\right), & v \in [1, \dots, N]\\ \boldsymbol{\overline{\mathcal{G}}}^{v \rightarrow 1} &= \mathrm{GSHead}\!\left(\{\boldsymbol{G}^{v}_{i}\}_{i=0}^{B}, \boldsymbol{I}^{v}\right), & v \in [1, \dots, N]. \label{eq:v2l95gs95head} \end{align}\tag{9}\]

Pose Head: For SPFSplatV2, the attended pose tokens \(\hat{\boldsymbol{g}^v}\) are fed into the pose head and further processed by a 3-layer MLP to predict the camera pose as a 10-dimensional representation [66]. The predicted pose representation is decomposed into translation and rotation for each view. The translation is represented using four homogeneous coordinates [66], while the rotation is encoded in a 6D format, capturing two unnormalized coordinate axes. These axes are normalized and combined via a cross-product operation to construct a full rotation matrix [67]. To compute the relative pose with respect to the reference view, the 10D pose representation is converted into a homogeneous transformation matrix \(\boldsymbol{P}^{v \to 1} \in \mathbb{R}^{4 \times 4}\). Following MASt3R, we make the pose head asymmetrical: \[\boldsymbol{P}^{ v \rightarrow 1} = \begin{cases} \mathrm{PoseHead}^1(\hat{\boldsymbol{g}^v}), & v = 1, \\ \mathrm{PoseHead}^2(\hat{\boldsymbol{g}^v}), & v \in [2, ..., V], \end{cases} \label{eq:v295pose95head}\tag{10}\] where \(\boldsymbol{P}^{v \to 1}\) is the estimated relative pose from \(\boldsymbol{I}^v\) to \(\boldsymbol{I}^1\).

For SPFSplatV2-L, the pose head follows the original VGGT design: the refined pose tokens \(\hat{\boldsymbol{g}}^v\) are subsequently processed by four additional self-attention layers and a linear projection to predict the camera parameters. \[\boldsymbol{P}^{ v \rightarrow 1} = \mathrm{PoseHead}(\hat{\boldsymbol{g}^v}), v \in [1,..., V]. \label{eq:v2l95pose95head}\tag{11}\]

We normalize the predicted poses relative to the first view, such that the first view has the canonical pose \([\mathbf{U} \mid \mathbf{0}]\), where \(\mathbf{U}\) is the identity matrix and \(\mathbf{0}\) is the zero translation vector.

3.3 Loss Functions↩︎

Image Rendering Loss: Our model is trained using ground-truth target RGB images as supervision. The training loss is defined as a weighted combination of the \(L_2\) loss and the LPIPS loss [68]: \[\mathcal{L}_{\text{render}} = \| \boldsymbol{I}^t - \hat{\boldsymbol{I}^t}\|_2 + \gamma \text{LPIPS}(\boldsymbol{I}^t, \hat{\boldsymbol{I}^t}), \label{eq:rendering95loss}\tag{12}\] where \(\boldsymbol{I}^t\) and \(\hat{\boldsymbol{I}}^t\) denote the ground-truth and rendered target images for \(t \in [N+1, V]\), and \(\gamma\) is a weighting factor that balances pixel-level accuracy and perceptual similarity.

Reprojection Loss: Existing approaches enforce pixel-aligned Gaussian prediction by constraining Gaussian locations along the input viewing rays [5], [6], [9], [11], [20], [21]. Meanwhile, canonical-space-based methods [7], [8] rely on ground-truth camera poses to guide the canonical 3D points (Gaussian centers). Both strategies ensure alignment between each pixel and its corresponding 3D point. However, since our model learns 3D Gaussian centers in a canonical space without known camera poses, the network lacks explicit geometric constraints to enforce pixel-aligned Gaussian representation.

A naive solution is to include context views in the image rendering loss (Eq. 12 ) by synthesizing images from them and computing the loss against their ground-truth counterparts. However, this leads to unstable training due to overfitting. Specifically, the network prioritizes improving the rendering quality of the first context view, as the 3D Gaussian space is defined in its camera coordinate, making its rendering independent of the learnable poses. Since the Gaussians from this view already capture sufficient scene information, the model suppresses the contribution of other context views by shifting their Gaussian centers away and adjusting camera poses, ultimately causing training collapse.

To address this issue, we instead employ a pixel-wise reprojection loss to jointly optimize 3D points and camera poses [12], [69]. Unlike purely image-based supervision, this reprojection loss enforces explicit geometric constraints, thereby reducing overfitting to the context views. Concretely, for each pixel \(\mathbf{p}^v_j\) in view \(v \in [1, N]\), we project the corresponding 3D Gaussian center \(\boldsymbol{\mu}_{j}^{v \rightarrow 1}\) from the canonical coordinate frame into the 2D pixel space using the estimated pose of view \(v\), and minimize the reprojection error: \[\begin{align} \mathcal{L}_{\text{reproj}} &= \sum_{v=1}^{N} \sum_{j=1}^{H\times W} \left| \mathbf{p}^v_{j} - \pi(\boldsymbol{K}^v, \boldsymbol{P}^{v \rightarrow 1}, \boldsymbol{\mu}_{j}^{v \rightarrow 1}) \right|, \end{align}\] where \(\pi\) denotes the camera projection function, \(\boldsymbol{K}^v\) the camera intrinsics of view \(v\), and \(\boldsymbol{P}^{v \rightarrow 1}\) the relative pose from view \(v\) to the canonical frame.

Different from SPFSplat, which applies reprojection loss to both context-only and context-with-target branches, SPFSplatV2 predicts a single set of context poses, avoiding redundant supervision and potential pose misalignment. This streamlined design leverages reprojection loss to enable more stable training and efficient optimization of pixel-aligned 3D Gaussians, without requiring ground-truth camera poses.

3.4 Multi-View Dropout↩︎

Unlike SPFSplat, which trains separate models for different numbers of context views, we use a single unified model. To improve generalization, we introduce a multi-view dropout strategy: for more than two context views, the leftmost and rightmost views are retained while intermediate views are randomly dropped during training. This encourages the network to handle flexible input configurations, improves robustness to varying numbers and spatial distributions of views at test time.

4 Experiments↩︎

We first introduce the experimental settings and implementation details in Sec. 4.1 and Sec. 4.2. We then evaluate the proposed framework under random initialization in Sec. 4.3. Subsequently, Sec. 4.4 presents results obtained with pretrained initialization. Finally, comprehensive ablation studies are conducted in Sec. 4.6 to analyze the contribution of each component of the proposed framework.

4.1 Experimental Settings↩︎

Datasets: In Sec. 4.3, we train our method on RealEstate10K (RE10K) [70] and evaluate it on both RE10K and out-of-domain ACID [71], following the official train-test splits used in prior work [5]–[7]. Following [7], evaluations on RE10K and ACID are conducted under varying camera overlaps, where image pairs are grouped into small (0.05–0.3), medium (0.3–0.55), and large (0.55–0.8) overlap ranges using overlap estimates from the pretrained dense matcher RoMa [72]. In Sec. 4.4, we train and evaluate our method on both RE10K and ACID for in-domain evaluation. We further evaluate cross-dataset generalization on ACID, DTU [73], DL3DV [74], and ScanNet++ [75]. To study scalability, we expand the training data beyond RE10K to include DL3DV, CO3Dv2 [76], and WildRGBD [77].

Baselines: For novel view synthesis, we compare to three groups of baselines: pose-required methods (pixelSplat [5], MVSplat [6], DepthSplat [30], YoNoSplat [29]), supervised pose-free methods (CoPoNeRF [17], Splatt3R [8], NoPoSplat [7]), and self-supervised pose-free methods (PF3plat [20], SelfSplat [21], SPFSplat [22], and AnySplat [43]). For pose estimation, we compare against SfM-based methods (SuperPoint [52] + SuperGlue [56], DUSt3R [32], MASt3R [23], VGGT [24]) and NVS-based methods (NoPoSplat, SelfSplat, PF3plat, SPFSplat and AnySplat).

11pt

Evaluation Protocol: For novel view synthesis, we adopt standard metrics: pixel-level PSNR, patch-level SSIM [78], and feature-level LPIPS [68]. For pose estimation, following prior works [7], [56], we report the area under the cumulative pose error curve (AUC) at thresholds of 5\(^\circ\), 10\(^\circ\), and 20\(^\circ\), where the pose error is defined as the maximum of the angular errors in rotation and translation.

During evaluation of novel view synthesis, target images are typically rendered with ground-truth poses [5], [6], [8], [17]. An alternative is to render using estimated target poses, as in PF3plat [20] and SelfSplat [21]. NoPoSplat [7] instead adopts an evaluation-time pose alignment (EPA) strategy, which optimizes the target pose during evaluation while keeping the reconstructed Gaussians fixed, so that the rendered image best matches the ground truth. This alignment decouples rendering quality from pose estimation accuracy, enabling direct assessment of Gaussian reconstruction. In contrast, rendering with estimated poses jointly evaluates reconstruction fidelity and the consistency between estimated poses and the learned Gaussians. Unless otherwise noted, we render with estimated poses for comprehensive evaluation and additionally report results with pose alignment for fair comparison to NoPoSplat.

4.2 Implementation Details.↩︎

Our method is implemented in PyTorch and leverages a CUDA-based 3DGS renderer with gradient support for camera poses. All models are trained on a single NVIDIA A100 GPU. Each training sample corresponds to a scene with context and target views, with the frame distance between context views gradually increased during training. The initial learning rate is set to \(1 \times 10^{-5}\) for the backbone and \(1 \times 10^{-4}\) for all other parameters, and LPIPS and reprojection losses are weighted 0.05 and 0.001, respectively. For SPFSplatV2, the encoder adopts a ViT-Large architecture with a patch size of 16, while the decoder is based on ViT-Base. For SPFSplatV2-L, the encoder is a ViT-Large from DINOv2 [79] with a patch size of 14. Training is performed at a resolution of 256 \(\times\) 256 and 224 \(\times\) 224 for V2 and V2-L, respectively.

4.3 Self-Supervised Pose-Free Learning from Scratch↩︎

Without geometric constraints or suitable initialization, the Gaussian center head predicts arbitrary 3D coordinates, providing no guarantee of a valid canonical structure and resulting in unstable or noisy gradients during early training. To stabilize optimization from random initialization, NoPoSplat and SPFSplat adopt a short warm-up phase with a point-cloud distillation loss from DUSt3R during early training. In contrast, we find that stable convergence can be achieved without such distillation by imposing a simple canonical constraint on the Gaussian center parameterization. Specifically, we derive a scalar depth value for each pixel by averaging its predicted 3D coordinates, and reproject it into 3D using the predicted camera poses and ground-truth intrinsics. The resulting aligned points serve as Gaussian centers throughout training, enforcing canonical consistency and enabling stable optimization from random initialization.

Empirically, as shown in Tab. ¿tbl:tab:nvs-from-scratch?, we make the following observations. 1) When trained from random initialization, NoPoSplat, despite benefiting from both DUSt3R distillation warm-up and ground-truth target pose supervision, fails to produce reliable pose estimates. In contrast, SPFSplatV2 optimizes geometry with self-predicted camera poses and achieves substantially stronger pose estimation performance, highlighting the effectiveness of self-supervised geometric optimization for learning camera poses from scratch. 2) Despite using no pretrained initialization, SPFSplatV2 and SPFSplatV2-L achieve significantly better novel view synthesis performance than existing pose-free methods that rely on pretrained geometric components, including SelfSplat, PF3plat, and AnySplat. This comparison is particularly noteworthy for AnySplat, which is trained on substantially larger mixed datasets and further benefits from VGGT-generated pseudo-label distillation throughout training. While AnySplat achieves higher pose estimation accuracy under these additional advantages, SPFSplatV2 and SPFSplatV2-L obtain superior NVS quality while maintaining competitive pose estimation performance using purely self-supervised optimization. 3) Furthermore, SPFSplatV2 and SPFSplatV2-L achieve comparable performance under random initialization despite their different architectural designs, suggesting that the proposed self-supervised training paradigm is robust and compatible with diverse model architectures. Collectively, these results demonstrate that effective self-supervised pose-free learning can emerge from random initialization without relying on pretrained geometric priors.

6.5pt

4.4 Self-Supervised Pose-Free Learning↩︎

Pretrained geometric priors have become a common component in both supervised and self-supervised 3DGS methods, serving either to stabilize optimization [7], [20], [21], [43] or to improve reconstruction quality [5], [6], [29], [30]. To facilitate a comprehensive and fair comparison with these approaches, we also investigate our framework under pretrained initialization. Importantly, these priors are used solely for initialization and are not involved in the subsequent self-supervised optimization. For SPFSplatV2, the encoder, decoder, and Gaussian center head are initialized from pretrained MASt3R [23] weights, while the pose head is initialized to approximate the identity rotation matrix for stable convergence. For SPFSplatV2-L, the encoder, decoder, pose head, and Gaussian center head are initialized from pretrained VGGT [24] weights. All remaining layers are randomly initialized. Since the pretrained Gaussian center heads already provide a stable canonical parameterization, the additional canonical constraint introduced in Sec. 4.3 is not required in this setting.

Novel View Synthesis: Quantitative results on RE10K and ACID are reported in Tab. ¿tbl:tab:rek95results? and Tab. 1. SPFSplatV2-L uses 224 \(\times\) 224 inputs, while all other methods operate at 256 \(\times\) 256. We make the following observations: 1) Despite being trained without ground-truth poses, SPFSplatV2 and SPFSplatV2-L consistently outperform state-of-the-art methods. Notably, both variants outperform our earlier SPFSplat baseline in most cases across different image overlap settings, with or without pose alignment, underscoring the effectiveness of the improved architecture. 2) While evaluation pose alignment generally improves performance, SPFSplatV2 without alignment still outperforms NoPoSplat with alignment, indicating that the jointly optimized poses are well aligned with the reconstructed Gaussians. 3) Between our variants, SPFSplatV2 achieves slightly higher PSNR on RE10K, while SPFSplatV2-L attains better LPIPS scores. On ACID, SPFSplatV2-L delivers the strongest overall results, likely benefiting from VGGT’s superior multi-view reconstruction capabilities and feature representations. 4) Compared with NoPoSplat, which is initialized from the same MASt3R weights, SPFSplatV2 achieves significantly better performance. This result highlights the effectiveness of the proposed self-supervised optimization compared with NoPoSplat’s reliance on ground-truth pose supervision during training. 5) Compared with AnySplat, which relies on both VGGT initialization and VGGT-generated pseudo labels throughout training, SPFSplatV2-L achieves significantly better performance across all image-overlap settings while using the same initialization but without pseudo-label distillation. We attribute this improvement mainly to the difference in training supervision. AnySplat supervises rendering only on context images and therefore receives no direct supervision for novel viewpoints. In contrast, our framework supervises target-view renderings with self-predicted poses, resulting in improved NVS performance.

Qualitative comparisons in Fig. 5 further demonstrate that our models reduce misalignment and recover more accurate geometry than baselines, even in challenging scenarios such as minimal input overlap or extreme viewpoint changes. Specifically, SPFSplatV2 improves structural accuracy and visual clarity compared to the original SPFSplat, while SPFSplatV2-L produces the highest overall rendering quality, capturing fine geometric details and textures more faithfully.

Cross-Dataset Generalization: To evaluate zero-shot generalization, we train on RE10K (indoor scenes) and test on ACID (outdoor), DTU (object-centric), DL3DV (outdoor), and ScanNet++ (indoor). As shown in Tab. ¿tbl:tab:cross-dataset?, both SPFSplatV2 variants generalize robustly across these diverse domains, consistently outperforming prior approaches. These datasets exhibit substantially different camera motions and scene types compared to RE10K, highlighting the strong out-of-domain generalization capability of our models, even under minimal image overlaps. Notably, in the RE10K\(\rightarrow\)ACID setting, SPFSplatV2 surpasses NoPoSplat and SPFSplat trained directly on ACID (Tab. 1) when evaluated with pose alignment. SPFSplatV2-L consistently outperforms SPFSplatV2 both with and without pose alignment.

Qualitative results in Fig. 6 show that both variants produce sharper and more geometrically accurate reconstructions than prior methods, with SPFSplatV2-L achieving the highest visual quality. These results demonstrate that, even without ground-truth poses, our framework effectively aligns 3D Gaussians with predicted camera poses, enabling robust generalization to out-of-distribution scenes.

Relative Pose Estimation. We evaluate relative pose estimation between input image pairs on RE10K, ACID, DL3DV, and ScanNet++, with results in Tab. ¿tbl:tab:pose95estimation?. All splat-based methods are trained on RE10K to evaluate generalization. Since VGGT does not natively support 224 \(\times\) 224 inputs, we resize and center-crop its input images to 224 \(\times\) 224 and pad the width to 518, as specified in [24]. SPFSplatV2-L uses 224 \(\times\) 224 inputs, while all other methods operate at 256 \(\times\) 256. SuperPoint + SuperGlue derives relative poses from Essential Matrices estimated from feature correspondences. DUSt3R, MASt3R, and NoPoSplat use PnP[44] with RANSAC [49], while PF3plat and VGGT directly predict poses. Our SPFSplat variants support two strategies: (i) direct regression through the pose head, and (ii) PnP with RANSAC applied to predicted 3D Gaussian centers.

As shown in Tab. ¿tbl:tab:pose95estimation?, both regression and PnP yield similarly strong performance, reflecting consistent alignment between estimated poses and reconstructed 3D points. Despite no geometry priors during training, SPFSplatV2 substantially outperforms MASt3R, its initialization model, and SPFSplatV2-L improves over VGGT in most cases. This demonstrates our framework’s ability to jointly optimize camera poses and 3D structure using only image-level supervision. Both SPFSplatV2 variants also significantly surpass the original SPFSplat, primarily due to masked attention improving pose alignment. On RE10K and ACID, our models achieve state-of-the-art results. On DL3DV and ScanNet++, which exhibit challenging camera motions, NoPoSplat achieves better performance, benefiting from ground-truth pose supervision during training. As shown in the scaling experiments below, this gap can be eliminated by scaling to larger training datasets.

Scaling to Larger Training Datasets. One of the key advantages of the proposed training paradigm is its ability to leverage larger and more diverse image collections without requiring camera pose supervision during optimization. To evaluate this property, we progressively scale the training data from RE10K alone to a 2-dataset setting (RE10K+DL3DV) and a 4-dataset setting (RE10K+DL3DV+WildRGBD+CO3Dv2) in Tab. ¿tbl:tab:scaling?. As shown in Tab. ¿tbl:tab:scaling?, increasing the scale and diversity of the training data consistently improves both novel view synthesis and pose estimation performance across all evaluation datasets. For SPFSplatV2, training on two datasets leads to substantial gains on DL3DV as well as the out-of-domain WildRGBD and ScanNet++ benchmarks compared to training on RE10K alone. Similar improvements are observed for SPFSplatV2-L, with performance continuing to increase as the training data expands from two to four datasets. The consistent gains across both architectures suggest that the improvements stem from the proposed training paradigm rather than a particular network design. These results demonstrate that the proposed framework scales effectively with data and can continuously benefit from larger and more diverse training collections. This scalability is particularly attractive for exploiting internet-scale image and video data, where accurate camera poses are often unavailable.

Geometry Reconstruction: As illustrated in Fig. 7, SPFSplatV2 and SPFSplatV2-L produce substantially higher-quality 3D Gaussian primitives than prior methods, even under large viewpoint changes between input pairs. Previous approaches often exhibit distorted structures or ghosting artifacts, whereas our models, trained without ground-truth poses, reconstruct more accurate 3D geometry and yield sharper renderings, reflecting improved Gaussian alignment across views. This improvement arises from the joint optimization of Gaussians and poses, which strengthens geometric consistency. Compared to SPFSplat, SPFSplatV2 achieves more precise structural reconstruction, particularly visible in the left windows, while SPFSplatV2-L further enhances overall Gaussian quality.

7pt

4.5pt

Extension to Multiple Views: Our method naturally extends to multiple input views. As shown in Tab. 2, novel view synthesis performance consistently improves with more context views. Both SPFSplatV2 and SPFSplatV2-L outperform the NoPoSplat and original SPFSplat, benefiting from enhanced architectures and the multi-view dropout strategy. With denser inputs, SPFSplatV2-L shows a clearer advantage, as its VGGT backbone, pretrained on multi-view data, provides stronger representations than MASt3R, which is limited to pairwise training. These results show that SPFSplatV2-L can better leverage additional views to enhance geometric consistency and reconstruction fidelity. Overall, the consistent gains with increasing context views highlight the flexibility and scalability of our framework for multi-view scenarios.

4.5 Efficiency↩︎

We compare the efficiency of our method to other approaches in Tab. 3 and Tab. 4. All methods use 256\(\times\) 256 inputs except SPFSplatV2-L, which uses 224\(\times\)​224 due to its backbone architecture with a patch size of 14.

Inference Efficiency: Tab. 3 reports parameter size, FLOPs, and runtime during inference, measured for reconstructing 3D Gaussians from two input images on an A6000 GPU. SPFSplatV2 achieves comparable model size, FLOPs, and runtime to NoPoSplat and SPFSplat, while providing substantial speedups of approximately 3.5\(\times\), 1.4\(\times\), 2.3\(\times\), and 27\(\times\) over pixelSplat, MVSplat, SelfSplat, and PF3plat, respectively. These gains stem from architectural differences: pixelSplat relies on a time-consuming epipolar transformer; MVSplat requires costly cost-volume construction; and both SelfSplat and PF3plat depend on separate pose-estimation modules to lift predicted depth into Gaussians, with PF3plat further incurring heavy local feature matching costs. In contrast, SPFSplatV2 reconstructs Gaussians directly in a canonical space using a feed-forward network, avoiding explicit geometric operations such as cost-volume construction. Compared to SPFSplatV2, SPFSplatV2-L introduces additional computational overhead, as a trade-off for superior reconstruction quality.

4pt

Training Efficiency: For a fair comparison, we evaluate the training efficiency of SPFSplat variants only against PF3plat and SelfSplat, as self-supervised pose-free methods require both context and target images during training, whereas other methods use only context images. The results are summarized in Tab. 4. For all methods, two images are used as context views and one as the target view. Training time and GPU memory usage are measured on an NVIDIA A100 and averaged per sample. PF3plat requires significantly larger FLOPs, time, and GPU memory during training. Thanks to masked attention, SPFSplatV2 reduces training FLOPs of SPFSplat by 12%, resulting in a 25% speedup and a 13% reduction in memory consumption. In contrast, SPFSplatV2-L incurs higher computational and memory costs, with a 46% increase in FLOPs, 30% longer training time, and 25% higher memory usage compared to SPFSplat, reflecting the trade-off for its superior reconstruction performance.

4.6 Ablation Analysis↩︎

5.5pt

Ablation on Learnable Pose Token: As shown in Tab. ¿tbl:tab:ablation?, removing the learnable pose token and reverting to SPFSplat’s global average pooling over feature maps leads to a marginal performance drop. This result suggests that the self-predicted pose estimation formulation is not tightly coupled to a specific pose prediction design. In particular, the learnable pose token enables compatibility with VGGT-style architectures while achieving slightly better performance.

Ablation on Masked Attention: As shown in Tab. ¿tbl:tab:ablation?, replacing masked attention with SPFSplat’s two-branch input design results in consistent performance degradation across both pose estimation and novel view synthesis, with a particularly significant impact on pose estimation accuracy. This effect is further illustrated in Fig. 8, where SPFSplat exhibits significant pose drift, whereas SPFSplatV2 produces camera poses that are better aligned with the reconstructed 3D geometry, leading to more accurate renderings.

Ablation on Intrinsic Embeddings: As shown in Tab. ¿tbl:tab:ablation?, removing intrinsic embeddings from the backbone reduces both reconstruction and pose estimation accuracy. This degradation is primarily attributed to increased scale ambiguity during joint optimization of 3D Gaussians and camera poses. Nevertheless, even without intrinsic embeddings, our method still outperforms NoPoSplat with intrinsic embeddings (Tab. ¿tbl:tab:rek95results?), achieving improvements of 0.5dB in PSNR when both evaluated with pose alignment.

Ablation on Reprojection Loss: Finally, removing the reprojection loss in Tab. ¿tbl:tab:ablation? and optimizing solely with the target-view rendering loss leads to a substantial degradation in both novel view synthesis and pose estimation performance. This highlights the importance of enforcing geometric consistency between 3D points and camera poses, and demonstrates that reprojection supervision is a key component for stable joint optimization and accurate reconstruction.

3.2pt

Ablation on Multi-view Dropout: As in Fig. 9, under the 5-view training setting, multi-view dropout randomly removes context views during training, exposing the model to diverse view configurations with up to five input views. In contrast, the baseline without multi-view dropout is trained using a fixed 5-view configuration. As a result, multi-view dropout enables the model to effectively generalize to varying numbers of input views at test time, with performance consistently improving as additional views are provided. Notably, the model continues to benefit from up to 10 input views during evaluation, despite never being trained with more than five views. These results demonstrate that multi-view dropout improves robustness to varying view counts and spatial distributions, leading to better generalization across multi-view settings.

Ablation on Ground-Truth Poses: In Tab. 5, we evaluate our framework’s ability to reconstruct geometry without ground-truth pose supervision by analyzing the effect of incorporating ground-truth poses during training in two settings: (b) rendering novel views using ground-truth poses, as in NoPoSplat, and (c) introducing a pose loss that penalizes the discrepancy between predicted and ground-truth poses, while still rendering with predicted poses. The pose loss combines a geodesic loss[80] for rotation and an \(L_2\) loss for translation. Since ground-truth poses are used only during training, the framework remains pose-free at inference.

Relative to setting (b), (a) SPFSplatV2 achieves improvements in both novel view synthesis and pose estimation. This can be attributed to the joint optimization of Gaussians and poses, which encourages better geometric alignment and more consistent feature learning. From (a) to (c), adding a pose loss improves pose accuracy but yields only marginal gains in novel view synthesis, underscoring the model’s capacity to reconstruct geometry without explicit pose supervision. These findings also suggest that high-quality novel view synthesis depends on factors beyond pose accuracy, such as occlusion, textureless regions, and extreme viewpoint changes, which may require generative priors or explicit 3D supervision.

Evaluation on In-the-Wild Data: We highlight the effectiveness of our model on mobile phone photos using SPFSplatV2 without intrinsic embeddings. The 3D geometry and rendered results in Fig. 10 demonstrate strong out-of-domain generalization, even under large viewpoint changes.

Failure Cases: As shown in Fig. 11, our method can produce blurred outputs or artifacts in occluded or texture-less regions, or under extreme viewpoint changes. Addressing these limitations may require stronger generative capabilities or larger training data.

5 Limitations and Future Work↩︎

Our method can be trained without ground-truth poses and scales effectively to large datasets, opening the possibility for future work to exploit more diverse data to further improve pose estimation and generalization. Nonetheless, it still benefits from the priors provided by supervised models such as MASt3R and VGGT, as evidenced by the performance drop when training from random initialization. Furthermore, since our approach is not generative, it cannot reconstruct unseen regions with high-fidelity textures. Incorporating generative models is a promising direction to address this limitation.

6 Conclusion↩︎

This paper presents SPFSplatV2, a self-supervised pose-free framework for 3D Gaussian splatting from sparse unposed views. By jointly optimizing camera poses and 3D Gaussian primitives through a unified backbone with masked attention, our approach achieves efficient and stable training as well as strong geometric consistency without requiring ground-truth poses. A reprojection loss is also incorporated with the conventional rendering loss to enforce pixel-aligned Gaussians. Extensive experiments on multiple datasets demonstrate that SPFSplatV2 and its larger variant SPFSplatV2-L establish new state-of-the-art results in novel view synthesis and cross-dataset generalization, while achieving strong relative pose estimation performance, even under challenging conditions of extreme viewpoint change and limited overlap. Importantly, the framework’s independence from ground-truth poses underscores its scalability to large and diverse real-world datasets, paving the way for future advances in scalable and generalizable 3D reconstruction.

References↩︎