1 Introduction↩︎

Visual localization is the problem of estimating the camera position and orientation in a 3D environment given a query image. Solving the localization problem is crucial in many applications such as autonomous driving [1], robot navigation [2], and augmented reality [3].

Visual localization approaches can be categorized by the underlying scene representation; see review in Section 2. Classical, structure-based methods [4]–[9] rely on explicit 3D models with keypoint descriptors, achieving accurate localization by matching 2D query image pixels to 3D model points. End-to-end learned approaches implicitly encode the scene through the weights of a neural network. These methods include absolute pose regression (APR) [10]–[15] and scene coordinate regression (SCR) methods [16]–[23]. These learned implicit representations are optimized exclusively for visual localization. [24], [25] use NeRF models [26] to refine the camera pose at test time through iterative rendering and pose adjustments, while [27]–[29] work toward directly estimating 3D-2D correspondences from NeRF.

Recently, 3D Gaussian Splatting (3DGS) [30] has gained attention as a promising scene representation for novel-view synthesis (NVS), offering fast training times and high-quality real-time rendering. Its growing popularity has spurred various studies exploring fundamental computer vision tasks within the 3DGS framework, including 3D segmentation [31], registration [32], and 3D editing [33]. Several methods have been proposed to address the localization challenge for 3DGS. However, they either only refine an initial pose [34]–[36], exhibit lower accuracy compared to other scene representations (6DGS [37]), or require optimizing a specific 3DGS model (GSplatLoc [38]).

In this paper, we propose to leverage the explicit scene representation of 3DGS for visual localization. Specifically, we introduce GSVisLoc, a generalizable deep-learning-based network for visual localization by matching 3DGS with 2D image features (see Figure 1). Given a query image and a 3D Gaussian Splatting scene representation, we use a KPConv-based encoder [39] to encode and downsample the Gaussians, producing features representing regions in 3D space. Additionally, we use an image encoder to extract features representing image patches from the query image. We then establish 3D-2D correspondences between the features, first at a coarse scale and then at a fine scale, enabling us to obtain an estimation for the camera pose. This is followed by an additional pose refinement step to achieve the final camera pose prediction.

Our method achieves accurate pose estimation that surpasses previous 3DGS-based localization methods. Moreover, GSVisLoc’s accuracy is competitive with state-of-the-art baselines on the 7-scenes dataset [16], [40]. Notably, our model generalizes to novel, unseen scenes without requiring any modifications or retraining for the 3DGS representation, and it eliminates the need for an image retrieval step.

In summary, our contributions include:

1. We introduce GSVisLoc, a deep neural network for visual localization for scenes represented using 3D Gaussian Splatting (3DGS).

2. Our approach requires no retraining or modifications to the underlying 3DGS representation, simplifying its deployment.

3. In contrast to structure-based and NeRF-based approaches, GSVisLoc eliminates the need for image retrieval during inference, allowing reference images to be discarded after training.

4. GSVisLoc achieves state-of-the-art performance among 3DGS-based methods across all evaluated datasets, with accuracy competitive with state-of-the-art methods on the 7-scenes dataset.

5. Most importantly, GSVisLoc generalizes to novel, unseen scenes through learned 3D-2D matching, retaining accuracy comparable to single-scene trained models.

2 Related work↩︎

Visual localization approaches can be grouped according to their underlying scene representation:
Structure-based localization methods [4]–[9] operate by establishing 3D-2D correspondences between 3D scene points and features in the query image. Camera pose is then calculated using a Perspective-n-Point (PnP) solver [41]–[43]. To streamline the matching process, an initial image retrieval step is used [44]–[47] to coarsely localize the query image, effectively narrowing the search space [48]–[50].

While this approach achieves accurate results, it requires substantial storage resources. Visual features [51]–[57] are typically extracted from a database of scene images to represent 3D points, and these features are then matched to features extracted from the query image to establish 3D-2D correspondences. To expedite inference, 3D descriptors are precomputed and stored alongside the scene model. This design, however, can lead to large memory footprints, complicating map updates and scalability [49]. Recent work has shown that storage requirements can be alleviated through compression and descriptor quantization [58], [59]. Complementary to compression, descriptor-free 2D–3D matching establishes pixel-to-point correspondences via learned geometric and photometric reasoning, avoiding persistent high-dimensional 3D descriptors [48]–[50].

Absolute Pose Regression (APR) methods [10]–[15] train models to directly predict the camera pose from a query image, eliminating the need for explicit 3D representations. Although APR methods offer high-speed performance, they typically lack the accuracy and generalization capabilities of structure-based approaches [60].

Relative Pose Regression (RPR) methods predict the transformation between image pairs instead of the absolute pose of a single query. This often improves generalization over APR [61]–[65], but RPR alone still falls short of the accuracy achieved by structure-based approaches.

Scene Coordinate Regression (SCR) methods [16]–[23] perform implicit 3D-2D matching by regressing the 3D scene coordinates directly from the query image. Like APR, these approaches use network parameters to encode the geometry of the scene [17]–[20], [22], but they are limited in representing large-scale scenes due to memory constraints. To address this limitation, more recent scene-agnostic SCR methods [21], [66] have decoupled the scene representation from the learned matching function, achieving scalability to larger scenes.

NeRF-based pose estimation methods [24], [25], [67], [68] rely on iterative rendering and pose adjustments, resulting in slow convergence and limited accuracy. NeFeS [25] improves APR pose estimation but struggles with SCR enhancements and suffers from a lengthy refinement time. HR-APR [69] accelerates the optimization process, yet each query still takes several seconds, even on a high-performance GPU. Other NeRF-based approaches, such as FQN [70], CrossFire [29], NeRFLoc [28], and NeRFMatch [27], enhance positioning accuracy by establishing 3D-2D correspondences. However, these methods require an image retrieval step during inference.

Following the recent introduction of 3D Gaussian-Splatting (3DGS) for novel view synthesis (NVS), several methods have explored its use also for pose estimation. 6DGS[37], which is closely related to our approach, achieves one-shot pose estimation by projecting rays from an ellipsoid surface, thus avoiding iterative processing. Although 6DGS uses 3DGS for visual localization, it achieves inferior accuracies compared to previous methods. GSplatLoc [38], a concurrent work, trains a Feature-3DGS model [71] in which each Gaussian is assigned a feature through training the Gaussian splatting model. This approach establishes correspondences by matching 3D to 2D features through a mutual nearest-neighbor search. It then uses the PnP algorithm to obtain an initial coarse pose estimate. The pose is subsequently refined iteratively to improve accuracy. In contrast to this method, our method uses the vanilla 3DGS model [30] and does not require optimizing a specific Gaussian model.

Recent works have utilized 3DGS specifically for pose refinement. For example, iComMa [34], and MCLoc [35], inspired by iNeRF [24], implements an iterative refinement process for camera pose estimation by inverting 3DGS. Similarly, GS-CPR[36] combines 3DGS with Mast3R [72], a powerful image matcher, to apply pose refinement.

Our method establishes precise 3D-2D correspondences and seamlessly integrates GS-CPR’s pipeline as a plug-and-play module for pose refinement, achieving superior performance over other 3DGS-based methods.

3 Method↩︎

3.1 Overview↩︎

We aim to establish point correspondences between a 3DGS scene representation and a 2D query image of the same scene. With these 3D-2D matches, a Perspective-n-Point (PnP) solver [41]–[43] can efficiently estimate the camera pose. To accomplish this, we propose GSVisLoc, a deep neural network for visual localization. Our network encodes the 3D Gaussians and the query image in latent feature spaces. It then matches similar features and uses those matches to estimate the camera parameters. We further enhance our results by refining the initial pose estimate. Details of the architecture, training process, and pose refinement are provided in the following sections.

3.2 GSVisLoc↩︎

Our network consists of a 3DGS encoder, a 2D encoder, and cross-modal matching modules to align features from both domains and establish 3D-2D correspondences. The 2D encoder encodes two types of features for coarse and fine matching. Refer to Figure 2 for a detailed schematic of our network architecture.

We use a 2D encoder to map image patches to a latent representation. Each latent vector encodes an image patch associated with a 2D location. Given a query image \(I\in \mathbb{R}^{H \times W \time 3}\), we use two levels of encodings: Coarse 2D features \(F_{m}^{c} \in \mathbb{R} ^ {N_c \times C_c}\), where \(N_c = \frac{H}{8} \times \frac{W}{8}\), and fine 2D features \(F_{m}^{f} \in \mathbb{R} ^ {N_f \times C_f}\), where \(N_f = \frac{H}{2} \times \frac{W}{2}\). \(C_c\) and \(C_f\) denote the encoding dimensions.

Since the number of Gaussians can be very large (typically 700K), it is impractical to match all of them with the 2D features. Therefore, we first filter Gaussians with low opacity value and then randomly select a subset of \(\hat{N}_g\) Gaussians.

We aim to match the locations of 3D Gaussians with image patches by aligning their features. Since the information in a single Gaussian is not rich enough to enable accurate matching, we construct an encoding incorporating also information from nearby Gaussians. To this end, we use KPFCNN [39], an encoder based on Kernel Point Convolution (KPConv) that applies convolution on a sphere. Additionally, KPFCNN incorporates downsampling layers that operate on a grid to reduce the number of 3D points to \(N_g\). The encoder receives the parameters of the Gaussians as input, including their opacity, radiance (represented by spherical harmonic coefficients), orientation, and scaling. The outputs are features \(F_{s} \in \mathbb{R}^{N_g \times C_c}\) associated with 3D points \(Q \in \mathbb{R}^{N_g \times 3}\), such that each feature represents a 3D region of Gaussians.

Next, given the scene and image features, we apply a series of interleaved self- and cross-attention layers. The self-attention layers operate independently on the 3D and 2D features, allowing each domain to refine its representations and capture local and global dependencies. The cross-attention layers enable interaction between the 3D and 2D features, aligning the two domains by focusing on shared information. This interleaved process allows for an iterative update of feature representations, facilitating precise 3D-2D correspondences.

Next, we compute pairwise cosine similarities between the coarse image features \(F_m^c\) and the 3DGS features \(F_s\), after we map them to a shared feature space, obtaining a matching score matrix \(\mathcal{S} \in \mathbb{R}^{N_{m} \times N_{s}}\). We then select matches whose scores exceed a threshold \(\theta_{c}\) while applying the mutual nearest neighbor (MNN) criterion to filter out potential outliers, yielding the final set of coarse correspondences: \[\mathcal{M}_{c} = \{(i, j) \mid (i, j) \in \operatorname{MNN}(\mathcal{S}), \, \mathcal{S}(i, j) \geq \theta_{c} \}. \label{eq:coarse-matches}\tag{1}\]

For each coarse match \(m_c = (i, j) \in \mathcal{M}_{c}\), we consider the fine-level feature \(F_m^f(i) \in \mathbb{R}^{w \times w \times D^f}\) corresponding to an image patch of size \(w \times w\) centered at the match location. We next apply a self-attention layer individually to each patch \(F_m^f(I)\) to enhance contextual information, while in parallel, we apply a linear layer to the coarse scene features to match the fine-level dimension \(D^f\).

Finally, we align the scene features with their corresponding fine image features. To that end, For each scene point, \(X_j\), we generate a heatmap expressing the probability that it aligns with each pixel near the image location \(x_i\).

The refined match location is then determined by taking an expectation over this probability distribution, resulting in the final fine-grained matches, denoted by \(\mathcal{M}_f\).

3.3 Training↩︎

For training, we use query images with known camera poses extracted from a Structure-from-Motion pipeline. We also use the output of this SfM pipeline to train our Gaussian Splatting scene representations. We train our model in three different pipelines: In Single-scene training, we train a model on the Gaussians and images of a single scene and evaluate the model on test images from the same scene (in-scene generalization). In Multi-scene training, we train a model on the Gaussians and images of a collection of scenes and evaluate the model on test images from the same set of scenes. Lastly, for Cross-scene generalization, we train a model on the Gaussians and images of a collection of scenes and evaluate the model on test images from novel unseen scenes.

We obtain ground-truth matches for training by projecting the 3DGS points \(X_s\) onto the query image using the ground-truth query camera pose and use their projected locations as their fine-level matches. For the coarse ground-truth matches, we compute a binary coarse association matrix \(M_{gt}\) as follows. For each 3D point \(X_j\), we use its projected location to assign it to the \(i^{th}\) \(8 \times 8\) patch to which it was matched, letting \(M_{gt}(i, j) = 1\) if \(X_j\) projects inside the 2D patch \(i\). Notice that a 3D point can match at most one image patch, yet a single image patch can match multiple 3D points.

The final loss consists of the losses for the coarse-level and the fine-level: \(L = L_c + L_f\):

To guide the coarse matching, we apply the log loss from [56]. This will work to increase the dual-softmax probability at the ground-truth matching locations in \(M_{gt}\). This loss is defined as \[\label{eq:coarse95matching95loss} L_c = - \frac{1}{|\mathcal{M}_{gt}|} \sum_{(i, j) \in \mathcal{M}_{gt}} \log (S(i, j)).\tag{2}\] where \(|\mathcal{M}_{gt}|\) is the total number of ground-truth coarse matches, and \(S(i,j)\) denotes the matching score between the image patch \(i\) and the 3D point \(j\).

Let \(X_j\) and \(x_j\) respectively denote a 3D point and its ground-truth match, obtained by projecting \(X_j\) using the ground-truth camera pose. We set the fine matching loss to minimize the pixel distance between the predicted location, \(\tilde{x}_j\), and the ground-truth location, \(x_j\), and, following [54], [56], weigh this distance by the total variance \(\sigma^{2}(j)\) of the corresponding heatmap to penalize more heavily deviations in matches that are more certain. The loss is given by \[\label{eq:fine95matching95loss} L_f = \frac{1}{|M_{f}|} \sum_{(i, j)\in M_f} \frac{1}{\sigma^2(j)} ||\tilde{x}_j - x_j||_2.\tag{3}\] where \(|\mathcal{M}_{f}|\) denotes the total number of predicted fine matches.

3.4 Pose refinement↩︎

We follow GS-CPR [36] for pose refinement. Starting from an estimated pose of the query image, we use the 3DGS representation to render both an image and a depth map. We next use MASt3R [72] to establish dense 2D-2D correspondences between the query image and the rendered image. The matched points in the rendered image are lifted to 3D using the rendered depth, the estimated pose, and camera intrinsics, creating 3D-2D correspondences for the query image. Finally, we apply PnP with RANSAC [73] to solve for the refined pose.

4 Experiments↩︎

4.1 Experimental setup↩︎

We conduct single-scene training on two well-established localization datasets. The 7-Scenes dataset [16], [40] consists of RGB-D images captured across seven unique indoor scenes (volumes ranging from \(1m^3\) to \(18m^3\)) that are challenging due to the presence of texture-less surfaces, motion blur, and occlusions. We follow the original train/test splits and use more accurate SfM pose annotations, as recommended by [19], [75], [80], for both our method and all baselines.

The Cambridge Landmarks dataset [10] contains handheld smartphone images from outdoor scenes, each characterized by significant exposure variations that complicate large-scale localization. Consistent with previous works [35], [38], we evaluate on King’s College, Old Hospital, Shop facade, and St Mary’s church (spanning \(875-5600m^{2}\)), following the original splits.

For cross-scene generalization, we use ScanNet++ V2 [81], a large-scale indoor dataset that couples high-quality (sub-millimeter laser scans) and commodity-level (registered images) geometry and color captures. Unlike setups that merely hold out a portion of training images along the same camera trajectory, ScanNet++ V2provides novel test views per scene, leading to more realistic and challenging conditions for cross-scene visual localization. The dataset encompasses large and varied indoor environments with numerous glossy and reflective surfaces, making accurate localization particularly difficult. We randomly sample 130 scenes for training, 7 for validation, and 15 for testing.

In line with previous works, we report median pose errors, specifically translation error in centimeters and rotation error in degrees. On the 7-Scenes dataset, we also report localization recall, which measures the percentage of query images localized with pose errors below specified thresholds—namely, \(5\text{cm}\) for the translation and \(5^\circ\) for the rotation.

We compared our method to MS-Trans. [15], DFNet [74], LENS [77], NeFeS [75], DSAC* [18], HACNet [20], ACE [19], SANet [21], DSM [78], NeuMap [66], InLoc[4], HLoc[9], PixLoc[79], CrossFire [29], NeRFLoc [28], NeRFMatch[27], GSplatLoc [38], DVLAD+R2D2[44], [76] and 6DGS [37]. Following [27], we categorize the methods into three groups: end-to-end methods, which include APR and SCR methods; hierarchical methods, where an initialized camera pose is estimated using an image retrieval step; and 3DGS-based methods. Additionally, we include the scene representation used for localization during testing. For experiments on Cambridge and 7-scenes, all the evaluations of the baseline methods were taken from previous works except for 6DGS [37], which we trained and evaluated using the official implementation. Since ScanNet++ V2 is a relatively new dataset, there are no publicly available baseline evaluations. Therefore, we compared our cross-scene generalization capabilities to both GSPlatLoc and our method in a single-scene training pipeline.

We use the first two blocks of ConvFormer [82] as the image backbone, initialized with ImageNet-1K [83] pre-trained weights¹. The feature dimensions are set to \(D^c = 512\) for the coarse matching and \(D^f = 128\) for the fine matching. For fine matching, a local window size of \(w = 5\) is used for image feature cropping. \(\theta_{c}=0.3\) the score threshold for the coarse matches. Query images are resized to \(480 \times 480\) in all experiments. We filter Gaussians with opacity values lower than 0.9 and then uniformly subsample 100K Gaussians. For the 3D backbone, we use a 3-stage KPFCNN [39] with output channels of {128, 256, 512} for each stage. We use the output of the final layer for encoding. The 3D-2D alignment module is composed of four interleaved self- and cross-attention layers. Both KPFCN and 3D-2D alignment modules are initialized with random weights. For pose refinement, we re-implemented GS-CPR, as the code is not yet publicly available. Unlike their approach, we use the original Gaussian splatting model [30] and omit the exposure-adaptive transformation. At inference, we apply three refinement iterations for each query image.

We train our models using the Adam optimizer [84] with a learning rate of \(\text{lr} = 0.0001\) and batch size \(\text{bs} = 1\) for 100 epochs. Our models are trained on a single Nvidia A40 GPU (48 GB). On a Quadro RTX 6000 and ScanNet++ V2 images at \(1752{\times}1168\), our average per-query time is \(2.82\) s (\(1.03\) s inference + \(1.79\) s refinement). The current implementation is not fully optimized and can be further accelerated. Further implementation details of the 3DGS are provided in the supplementary material.

We determined the values of hyper-parameters by running our models on the validation split from the 7-scenes datasets. These hyper-parameters include the learning rate, batch size, number of interleaved self- and cross-attention layers in the 3D-2D alignment module, and the number of layers used by the 3D encoder.

4.2 Single-scene training↩︎

As shown in Table ¿tbl:tab:loc957sc?, our method outperforms other 3DGS-based approaches and achieves higher rotation and translation accuracy than APR and NeRF-based methods. It also performs on par with SCR methods, though it is slightly outperformed by the structure-based DVLAD+R2D2 [44], [76].

As shown in Table 1, our approach surpasses GSplatLoc, a concurrent 3DGS-based method, in rotation accuracy while achieving comparable translation accuracy. It further outperforms APR methods by a significant margin in both rotation and translation accuracy. However, it trails behind the state-of-the-art method HLoc, particularly in translation accuracy. We believe this is due to the relatively low rendering quality of 3DGS on the Cambridge Landmarks dataset (see supplementary material). Improved 3DGS representations optimized for such challenging scenes may help bridge this gap.

In Figure 3, we provide qualitative results demonstrating the accuracy of the pose estimation achieved with our method both before and after the refinement stage. Each example includes the ground-truth (GT) query image and rendered images from the initially estimated pose and the final refined pose.

The initial pose estimates align reasonably well with the GT, though some misalignments remain. After refinement, the final pose rendering shows improved alignment, with details that closely match the GT image. This refinement step corrects subtle orientation and translation discrepancies.

4.3 Multi-scene training↩︎

A multi-scene model can encode information from multiple scenes in a single model while retaining accuracy similar to single-scene training. In addition, a multi-scene model allows for lower resource usage at inference. We tested the ability of our model to encode several scenes simultaneously. To that end, we trained our model on training images from all the scenes in the 7-scenes dataset and tested it on test query images from the same scenes. Since the size of the training data for this experiment is large, we train GSVisLoc without the fine-level matching (coarse-level only) and compare it to single-scene models with fine-level matching. The results in Table 2 indicate that, despite training across multiple scenes at coarse-level, we observe only a slight reduction in prediction accuracy and even an improvement in the Stairs scene.

max width=

4.4 Cross-scene generalization↩︎

We next tested our model for cross-scene generalization. We trained the model on multiple scenes from the ScanNet++ V2 dataset and tested it on novel, unseen scenes from the test split. We compared the cross-scene model to GSPlatLoc and to our method trained in the single-scene training pipeline. We note that these two baselines, unlike our cross-scene model, were trained specifically on the tested scenes. Due to shortage of resources, we trained this model without the fine-level matching, as in the multi-scene setup. As shown in Table  ¿tbl:tab:generalization?, our cross-scene model outperforms GSPlatLoc and achieves comparable results to our single-scene pipeline.

4.5 Ablations↩︎

We conducted ablation studies to demonstrate the importance of different components of our model. We tested our model without (a) a 3D encoder, (b) the fine-tuning of a 2D encoder, (c) a 3D-2D alignment module, and (d) fine-level matching. We report the results before pose refinement in Table 3. For (a), when the 3D encoder was omitted, we sampled \(N_g\) 3D points and used a single linear layer to align the feature dimensions between the Gaussians and the query image. In this setup, the model was unable to extract a sufficient number of matches for reliable camera pose estimation, highlighting the necessity of encoding a 3D region rather than relying on individual Gaussians. Training a model without either fine-level matching or fine-tuning the 2D encoder hurts the model’s accuracy, demonstrating their importance. Surprisingly, removing the 3D-2D alignment module (i.e., using dual-softmax directly on the encoder outputs, without applying self and cross-attention layers) had minimal impact on the results in a single-scene training setup, indicating that the encoders effectively extract corresponding 3D and 2D features. However, in the cross-scene generalization setup, omitting the 3D-2D alignment module led to near-random pose estimates.

5 Conclusion↩︎

In this work, we present GSVisLoc, a generalizable visual localization method for a 3D Gaussian Splatting (3DGS) scene representation, whose task is to estimate the camera position and orientation in a 3D environment given a query image. Our method, which uses deep learning to establish 3D-2D correspondences enables effective localization with 3DGS as the sole scene representation, eliminating the need for reference or training images during inference. Our method achieves competitive localization accuracy on standard benchmarks that include images from both indoor and outdoor scenes. The coarse-to-fine matching strategy, coupled with a 3DGS-based pose refinement step, allows for precise pose estimation. Most importantly, our method is trainable, enabling cross-scene generalization with almost no loss of accuracy. By that, our work opens avenues for visual localization for 3DGS scene representation, without reliance on additional reference data. Still, our results highlight the importance of improving 3DGS scene representation on outdoor scenes.

Acknowledgments↩︎

Research was partially supported by the Israeli Council for Higher Education (CHE) via the Weizmann Data Science Research Center, by the MBZUAI-WIS Joint Program for Artificial Intelligence Research and by research grants from the Estates of Tully and Michele Plesser and the Anita James Rosen and Harry Schutzman Foundations

Below, we provide additional details about our implementation and examples of challenging scenes from the ScanNet++ dataset in Figure 4. As shown, the scenes are large and diverse, including extensive texture-less areas, which demonstrates the generalization ability of our cross-scene model.

Our code is still a work in progress. We will publish it after finishing to clean and refactor it for easy use. All of the data we used is publicly available. We will also release our pre-trained models.

6 Gaussain Splatting Implementation Details↩︎

We use the pre-built COLMAP reconstructions from [80] for the 7scenes dataset and the reconstructions provided in HLoc toolbox [9] for the Cambridge landmarks dataset. We train all the scenes using the vanilla 3DGS [30], for 30k iterations using the default parameters, in Table ¿tbl:tab:gs95psnr? we report the per-scene PSNR scores for our trained models on the training images. Notably, the rendering quality of outdoor scenes is inferior compared to indoor scenes, which might explain the degradation in our pose estimation accuracy.

To effectively train a 3DGS model for outdoor scene reconstruction, we focus on reconstructing static elements such as buildings, fences, and signs. This approach addresses real-world challenges like varying lighting conditions, dynamic objects, and distant regions. To mitigate these issues, we use a pre-trained semantic segmentation model [85] to mask out sky regions and moving objects, including pedestrians and vehicles. These elements, which constitute only a small portion of the captured images, are excluded from the loss function during training, resulting in more accurate scene reconstruction. For this purpose, we utilize pre-computed segmentation maps provided by [27], generated using the method described in [85].

References↩︎