Consistent cross-view understanding is essential for spatial intelligence, as it enables agents to recognize the same scene from different viewpoints [1]–[4]. Recent foundation models have greatly advanced this capability under small viewpoint changes. Semantic foundation models such as DINO [5]–[7] learn rich dense representations that capture object-level and region-level semantics, while 3D foundation models such as Depth Anything 3 (DA3) [8] recover plausible scene structure and camera geometry. However, these capabilities are still largely developed from observations within a single view or across relatively small viewpoint changes [9]–[12]. When the viewpoint gap becomes extreme, such as between ground-view imagery and overhead satellite-view imagery, current visual foundation models still struggle to achieve consistent cross-view understanding. For example, semantic representations may be reliable within each individual view but fail to establish consistent correspondences across drastically different viewpoints, while structure and geometry that appear locally plausible may not remain consistent across such views. Addressing this limitation is particularly important for spatial intelligence, as it allows models to reason despite drastic appearance variations and develop more generalizable scene understanding.

Humans naturally develop such an ability. During self-localization, for example, when observing a scene from the ground, we can associate ground-level observations with an overhead map, forming a consistent understanding of the same scene across drastically different viewpoints. Inspired by this, we investigate whether cross-view localization can serve as a learning framework for acquiring consistent cross-view understanding under extreme viewpoint changes, including stable semantics, reliable structure, and transferable geometry.

Cross-view localization naturally provides a promising pathway toward consistent cross-view understanding under extreme viewpoint changes, as it requires a model to align ground-view imagery with geo-referenced satellite-view imagery to estimate camera poses [13]–[17]. With the rapid development of visual foundation models, recent studies have shown that establishing sparse correspondences across drastically different viewpoints is becoming increasingly feasible [18], [19]. By exploiting the rich 2D representations provided by foundation models, these methods typically formulate localization as a two-stage pipeline, where sparse 2D correspondences are first established between ground-view and satellite-view images, and the camera pose is then estimated from the predicted matches. This sparse matching paradigm resembles human self-localization, where a few distinctive landmarks can often be sufficient for accurate localization. However, the two-stage pipeline decouples correspondence from pose, overlooking the fact that cross-view localization is fundamentally a 3D reasoning problem, in which correspondences and the camera pose are intrinsically coupled [20]. To be more specific, a correspondence is meaningful only when it is supported by a valid pose hypothesis, while a pose can be reliably determined only when the induced correspondences are globally consistent. Such decoupling limits the learning potential of cross-view localization for acquiring consistent cross-view understanding under extreme viewpoint changes. In particular, it poses three technical limitations for learning stable semantics, reliable structure, and transferable geometry, as shown in Fig. 1 (a).

First, existing formulations lack explicit 3D grounding, which limits their ability to support structure learning [18], [19], [21], [22]. Most methods learn correspondences directly in the 2D image domain, without requiring the model to reconstruct, infer, or reason about the underlying 3D scene structure. As a result, localization is often solved primarily through appearance matching, providing limited incentive for the model to learn reliable scene structure.

Second, most methods enforce strict point-wise matching across views [18], [19], encouraging the model to distinguish visually similar objects in an exclusive manner, which can hinder the learning of stable semantics. Cross-view localization naturally exhibits many-to-many semantic consistency. For example, a house in the ground-view image should not be semantically associated with only one house in the satellite-view image, but rather with multiple semantically consistent house instances. Enforcing rigid one-to-one point-wise matching can bias the representation toward instance discrimination and weaken semantic consistency across viewpoints.

Third, existing learning objectives provide limited supervision for learning transferable geometry. They often treat localization as absolute pose prediction, rather than evaluating which pose hypothesis best explains cross-view geometric consistency [21]–[25]. In this formulation, ground-truth poses serve only as final targets, offering limited guidance on why a pose is plausible or how it differs from alternative hypotheses. This limitation becomes more pronounced under weak supervision, where previous methods often decouple orientation and translation to exploit partially available pose labels [26]–[28]. Such decoupling ignores the inherent geometric dependency between orientation and translation, preventing the model from evaluating the plausibility of coupled pose hypotheses as a whole. Without explicitly reasoning about coupled pose hypotheses through cross-view consistency, existing methods struggle to learn coherent camera geometry and often fail to generalize beyond the trained ground-to-satellite setting.

To address these limitations, we propose CROSS, a unified framework for developing consistent cross-view understanding and learning stable semantics, reliable structure, and transferable geometry from cross-view localization. Rather than treating localization as independent 2D matching followed by pose estimation, CROSS introduces 3D-grounded alignment and structure-aware matching to evaluate the cross-view consistency of pose hypotheses, and learns through hypothesis ranking, as shown in Fig. 1 (b).

First, CROSS performs 3D-grounded alignment to learn reliable structure. Given a ground-view image, sparse visual representations are lifted into 3D using predicted metric-scale depth and then projected onto the satellite-view image under each pose hypothesis. This establishes a 2D-3D-2D alignment pathway, where a hypothesis can be verified only when the predicted ground-view structure is consistent with the metric layout of the satellite view. Therefore, structure is not learned merely to be locally plausible within a single viewpoint, but to be cross-view consistent as a representation of the same world.

Second, CROSS learns stable cross-view semantics through structure-aware matching. Instead of enforcing brittle one-to-one correspondences between viewpoints, CROSS aggregates matching evidence over the holistic structure induced by each pose hypothesis. This allows the model to evaluate whether a set of scene elements forms a globally plausible alignment, rather than requiring every local feature to match independently. For example, a house in the ground view may be matched to a different but semantically similar house in the satellite view under an incorrect pose hypothesis. Such a local ambiguity is not overly penalized, since the model can accumulate contradictory evidence from the holistic scene structure. As a result, semantic representations are encouraged to remain stable across extreme viewpoints, rather than collapsing into instance-specific biases.

Third, CROSS learns transferable geometry by ranking pose hypotheses according to cross-view consistency. Each pose hypothesis specifies the observer-dependent camera geometry and determines how the predicted structure is projected across viewpoints. The model therefore learns not only what the target pose is, but also why it is plausible, since a valid pose should produce semantically, structurally, and geometrically consistent alignment across views. Under full supervision, the ground-truth pose is ranked above perturbed hypotheses. Under weak supervision, the missing pose dimensions are treated as latent variables within the same hypothesis space and optimized through their consistency with the known pose dimensions. This unified formulation preserves the coupling between orientation and translation, thereby supporting both fully and weakly supervised training. By ranking hypotheses through cross-view consistency, CROSS acquires transferable geometry that generalizes beyond the trained viewpoint transformation.

The contributions of this work are summarized as follows:

• Beyond pose estimation, we revisit cross-view localization as a learning framework for acquiring consistent cross-view understanding under extreme viewpoint changes, including stable semantics, reliable structure, and transferable geometry.

• We propose CROSS, a unified cross-view localization framework that introduces 3D-grounded alignment and structure-aware matching to support reliable structure learning and stable semantic representation.

• We introduce a pose hypothesis ranking objective for transferable geometry learning. By ranking pose hypotheses according to their induced cross-view consistency, CROSS naturally supports both fully and weakly supervised settings and learns transferable geometry beyond the trained transformation.

• Extensive experiments demonstrate that CROSS achieves state-of-the-art localization performance and effectively learns stable semantics, reliable structure, and transferable geometry from cross-view localization.

1 Related Work↩︎

We discuss related methods in three aspects: cross-view localization, semantic learning, and structural and geometric learning.

Cross-View Localization. Cross-view localization requires a model to align ground-view imagery with geo-referenced satellite-view imagery to estimate camera poses, thus providing a promising pathway toward consistent cross-view understanding under extreme viewpoint changes. Early studies commonly formulated this problem as image retrieval, where a ground-view query is matched against a database of satellite-view images using global descriptors or learned cross-view embeddings [13]–[17], [29]–[33]. Recent fine-grained localization methods move beyond global retrieval and aim to estimate 3-DoF camera poses, including translation and orientation, by establishing dense or sparse alignments between ground-view and satellite-view images [18], [19], [22]–[25], [34].

Dense Matching Methods. Motivated by the strong perception capability of bird’s-eye-view (BEV) representations in 3D scene understanding [35]–[38], early fine-grained cross-view localization methods commonly reduce the drastic viewpoint gap by transforming ground-view observations into dense BEV or map-aligned representations, followed by dense alignment with satellite imagery [23]–[25], [39]–[41]. These methods provide a spatially organized intermediate representation between ground and aerial views, but they also inherit the limitations of dense BEV representations. Perspective-to-BEV transformation may compress height information, blur fine-grained details, and introduce redundant features, making alignment computationally expensive and less interpretable.

Sparse Matching Methods. Recently, sparse matching methods have emerged as a more favorable alternative to dense BEV alignment. Rather than correlating dense feature maps, these methods establish sparse correspondences between ground-view and satellite-view images and estimate the camera pose from the predicted matches. FG\(^2\) formulates fine-grained cross-view localization as feature matching between a ground-derived point plane and an aerial point plane, improving interpretability by tracing which local features contribute to pose estimation [18]. Loc\(^2\) further learns ground-to-satellite correspondences with monocular depth priors, enabling pose estimation from depth-lifted local correspondences [19]. These methods mark an important shift from implicit dense matching toward explicit and interpretable sparse matching.

Nevertheless, most sparse matching methods still separate correspondence establishment from pose estimation, rely on strict point-wise matching, and learn from an absolute objective. As a result, despite their improved localization effectiveness and efficiency, they do not fully exploit the potential of cross-view localization for acquiring consistent cross-view understanding under extreme viewpoint changes. In contrast, CROSS introduces 3D-grounded alignment to learn reliable structure, structure-aware matching to develop stable semantics, and hypothesis ranking to acquire transferable geometry.

Semantic Learning. Recent vision foundation models have demonstrated remarkable capabilities in semantic representation learning through large-scale pretraining. Self-supervised methods such as DINO [5] show that object- and part-level semantics can emerge from visual pretraining without dense annotations, while DINOv2 [6] and DINOv3 [7] further scale this paradigm to obtain strong general-purpose features across diverse visual domains. Vision-language models also learn transferable semantics from image-text supervision and enable broad zero-shot recognition [42]–[49]. These foundation models provide powerful semantic priors and have been widely adopted as generic visual encoders for downstream tasks.

However, the semantics learned by these models are not explicitly constrained to remain consistent across extreme viewpoint changes. Most pretraining objectives operate on single images, image-text pairs, or visually related views, rather than paired observations of the same 3D scene from drastically different viewpoints. As a result, their features may capture strong intra-view semantics while still exhibiting inconsistent cross-view correspondences between ground-view and satellite-view imagery. In contrast, CROSS uses cross-view localization itself as supervision for semantic learning, encouraging semantic representations to remain stable across viewpoints and directly support cross-view reasoning.

Structural and Geometric Learning. Recent vision foundation models have also made rapid progress in learning scene structure and camera geometry from large-scale data. Monocular depth models, such as Depth Anything [50], Depth Anything V2 [51], and UniDepthV2 [52], learn strong depth priors that generalize across diverse image domains, providing a practical foundation for single-view structure understanding. Recent models further move beyond independent depth prediction toward unified 3D perception. DUSt3R [9] formulates unconstrained stereo reconstruction as dense point-map regression, enabling depth, matching, and camera estimation without known calibration or poses. Following this direction, MASt3R [10] strengthens image matching with 3D-grounded representations, and VGGT [11] unifies key geometric predictions, including camera parameters, depth maps, point maps, and point tracks, in a feed-forward framework. MapAnything [12] further extends this trend toward universal metric 3D reconstruction by accepting flexible geometric inputs, while Depth Anything 3 [8] predicts spatially consistent geometry by introducing a unified depth-ray representation. These works show that large-scale pretraining can substantially enhance unified spatial understanding.

Despite these advances, existing foundation models are mainly designed to recover structure and geometry within a single viewpoint or across relatively small viewpoint changes. Their predictions can be locally plausible under standard perspective geometry, but are not explicitly optimized to explain consistency across extremely different viewpoints, such as ground-view and satellite-view imagery. In contrast, CROSS learns reliable structure and transferable geometry from cross-view localization by evaluating cross-view consistency of hypotheses, thereby generalizing beyond the trained viewpoint transformation.

2 Proposed Method↩︎

Given a ground-view image \(\mathbf{I}_g\) with an initial noisy 3-DoF camera pose \((\tilde{x}, \tilde{y}, \tilde{\theta})\), where \((\tilde{x}, \tilde{y})\) denotes the horizontal translation and \(\tilde{\theta}\) denotes the orientation angle, together with a geo-referenced satellite-view image \(\mathbf{I}_s\), CROSS estimates the refined 3-DoF pose \((\hat{x}, \hat{y}, \hat{\theta})\) of the ground-view camera. The overall framework is illustrated in Fig. 2 and consists of four main components. First, spatial representation learning predicts scene structure and semantic embeddings. Second, 3D-grounded alignment introduces a 2D-3D-2D pathway that encourages the model to learn reliable structure. Third, structure-aware matching aggregates consistency evidence over the holistic structure to guide the learning of stable semantics. Finally, hypothesis ranking enables CROSS to learn from both fully and weakly supervised training within a unified framework and develop transferable geometric understanding.

2.1 Spatial Representation Learning↩︎

The spatial representation learning stage aims to extract rich semantic and structural features from both ground-view and satellite-view images.

A frozen DINO encoder [6], [7] \(\Phi_{\mathrm{DINO}}\) first extracts intermediate visual tokens, which are then processed by a shared DPT [53] decoder \(\Phi_{\mathrm{DPT}}\) to produce multi-scale feature maps at strides \(\ell \in \{1,2,4,8\}\): \[\mathbf{F}_g^\ell = \Phi_{\mathrm{DPT}}(\Phi_{\mathrm{DINO}}(I_g)), \quad \mathbf{F}_s^\ell = \Phi_{\mathrm{DPT}}(\Phi_{\mathrm{DINO}}(I_s)).\]

Unlike previous methods, we employ a shared decoder \(\Phi_{\mathrm{DPT}}\) across viewpoints for both structural and semantic learning, encouraging the model to acquire a robust and generic spatial representation.

2.1.1 Structural Learning↩︎

To capture the structural layout of the scene, CROSS applies a structure head to ground-view features and predicts metric-scale depth maps.

For each scale \(\ell \in \{1,2,4,8\}\), the corresponding ground-view feature map \(\mathbf{F}_g^\ell \in \mathbb{R}^{C \times H \times W}\) is processed by a Multi-Layer Perceptron (MLP) to produce depth distribution logits \(\tilde{\mathbf{D}}_{g}^\ell \in \mathbb{R}^{D \times H \times W}\): \[\tilde{\mathbf{D}}_g^\ell = \Phi_{\mathrm{depth}}(\mathbf{F}_g^\ell),\] where the \(D\) channels of \(\tilde{\mathbf{D}}_{g}^\ell\) represent logits distributed over \(D\) depth bins \(\mathbf{b}=[b_1,\dots,b_D]^\top\) that are uniformly spaced in the log-depth domain.

The depth logits are then converted into a discrete probability distribution with a softmax operation, and the final predicted depth is computed as the expected value over these bins, yielding a depth map \(\mathbf{D}^\ell_{g} \in \mathbb{R}^{H\times W}\) at stride \(\ell\): \[\mathbf{D}_g^\ell(i,j) = \mathrm{softmax}(\tilde{\mathbf{D}}_g^\ell(i,j)) \cdot \mathbf{b}.\]

To model depth uncertainty, a separate MLP predicts the log-variance for each pixel, producing a log-variance map \(\mathbf{\Sigma}^\ell_{g} \in \mathbb{R}^{H \times W}\): \[\mathbf{\Sigma}_g^\ell = \Phi_{\mathrm{unc}}(\mathbf{F}_g^\ell),\] where a higher log-variance indicates greater depth uncertainty.

Both \(\Phi_\mathrm{depth}\) and \(\Phi_\mathrm{unc}\) share weights across all scales, allowing multi-scale structural cues to jointly guide depth estimation and produce a more reliable scene structure representation.

2.1.2 Semantic Learning↩︎

To develop stable semantics for cross-view localization, CROSS employs a semantic head that predicts both semantic embeddings and reliability scores. This dual output is motivated by the fact that transient or weakly grounded objects, such as cars and pedestrians, are often unreliable for establishing robust cross-view associations.

An MLP \(\Phi_{\mathrm{sem}}\) first transforms both ground-view and satellite-view feature maps into semantic embedding maps: \[\mathbf{S}_g^\ell = \Phi_{\mathrm{sem}}(\mathbf{F}_g^\ell),\qquad \mathbf{S}_s^\ell = \Phi_{\mathrm{sem}}(\mathbf{F}_s^\ell),\] where \(\mathbf{S}_g^\ell,\mathbf{S}_s^\ell \in \mathbb{R}^{C\times H \times W}\) for notational simplicity. In practice, the ground-view image and satellite-view image may have different spatial resolutions.

A separate MLP \(\Phi_{\mathrm{rel}}\) predicts reliability maps that estimate how trustworthy each pixel is for cross-view alignment: \[\mathbf{R}_g^\ell = \Phi_{\mathrm{rel}}(\mathbf{F}_g^\ell),\qquad \mathbf{R}_s^\ell = \Phi_{\mathrm{rel}}(\mathbf{F}_s^\ell),\] where \(\mathbf{R}_g^\ell,\mathbf{R}_s^\ell \in \mathbb{R}^{H \times W}\).

Crucially, both \(\Phi_{\mathrm{sem}}\) and \(\Phi_\mathrm{rel}\) share weights across all scales and both viewpoints, encouraging CROSS to learn a consistent and generic cross-view semantic representation.

2.2 3D-Grounded Alignment↩︎

To address the lack of 3D grounding in existing formulations, we propose 3D-grounded alignment, which introduces a 2D-3D-2D pathway that requires the model to reason about scene structure.

Since ground-view images cover only a limited region compared with satellite-view images, we construct the spatial graph primarily from the ground-view image to reduce the introduction of irrelevant outliers. Based on the multi-scale ground-view reliability maps \(\mathbf{R}_g^\ell\) and semantic embedding maps \(\mathbf{S}_g^\ell\), we extract a set of \(N\) reliable nodes. For each \(\ell \in \{1,2,4,8\}\), we apply non-maximum suppression to \(\mathbf{R}_g^\ell\) and select the top-\(N/4\) locations. All selected locations are mapped to the original ground-view image resolution, yielding 2D node coordinates \(\{\mathbf{x}_i\}_{i=1}^{N}\), where \(\mathbf{x}_i\in\mathbb{R}^{2}\).

For each selected node \(i\), we sample its semantic embedding \(\mathbf{s}_i \in \mathbb{R}^C\), reliability score \(r_i \in \mathbb{R}\), and predicted metric depth \(d_i \in \mathbb{R}\): \[\mathbf{s}_i = \mathbf{S}^\ell_g(\hat{\mathbf{x}}_i), \quad r_i = \mathbf{R}^\ell_g(\hat{\mathbf{x}}_i), \quad d_i = \mathbf{D}^\ell_g(\hat{\mathbf{x}}_i),\] where \(\hat{\mathbf{x}}_i\) denotes the coordinates scaled to the corresponding feature resolution for sampling. This gives the ground-view spatial graph \(\mathcal{G}_g\): \[\mathcal{G}_g = \{(\mathbf{x}_i, \mathbf{s}_i, r_i)\}_{i=1}^N.\]

Given a ground-view camera pose hypothesis \(\mathbf{H}=(\mathbf{R}, \mathbf{t})\), generated as described in Sec. 2.4, each node is first back-projected into the 3D world coordinate frame using its predicted depth \(d_i\): \[\mathbf{X}_i = \mathbf{R}(d_i \mathbf{K}_g^{-1}[\mathbf{x}_i^\top, 1]^\top)+\mathbf{t},\] where \(\mathbf{K}_g\) denotes the ground-view camera intrinsic matrix. The 3D point is then projected onto the satellite-view image plane: \[\mathbf{y}_i = \Pi_{\mathrm{sat}}(\mathbf{X}_i),\] where \(\Pi_{\mathrm{sat}}\) denotes the projection mapping from world coordinates to satellite pixel coordinates. This yields the 2D satellite-view graph \(\mathcal{G}_s(\mathbf{H})\) induced by hypothesis \(\mathbf{H}\): \[\mathcal{G}_s(\mathbf{H}) = \{(\mathbf{y}_i, \{\mathbf{s}_i^\ell\}, \{r_i^\ell\})\}_{i=1}^N,\] where the node at the coordinate \(\mathbf{y}_i\) has multi-scale semantic embeddings \(\{\mathbf{s}_i^\ell\}\) and reliability weights \(\{r_i^\ell\}\) sampled from the multi-scale satellite-view semantic embedding maps \(\{\mathbf{S}_s^\ell\}\) and reliability maps \(\{\mathbf{R}_s^\ell\}\), and \(\ell \in \{1,2,4,8\}\).

Through this 2D-3D-2D pathway, sparse cross-view correspondences between \(\mathcal{G}_g\) and \(\mathcal{G}_s(\mathbf{H})\) are explicitly grounded in the predicted 3D scene structure. A pose hypothesis can therefore be evaluated according to whether the ground-view structure produces a spatially consistent alignment with the satellite view.

2.3 Structure-Aware Matching↩︎

To overcome the limitations of point-wise matching, we employ structure-aware matching to evaluate cross-view consistency by aggregating evidence over the holistic scene structure. This encourages the model to learn semantic representations that remain stable across viewpoints.

Given a pose hypothesis \(\mathbf{H}\), we evaluate its plausibility based on the ground-view spatial graph \(\mathcal{G}_g\) and the hypothesis-induced satellite-view spatial graph \(\mathcal{G}_s(\mathbf{H})\) obtained from 3D-grounded alignment. For the \(i\)-th ground-view node, we compute the cosine similarity between its semantic embedding \(\mathbf{s}_{i}\) and the corresponding satellite-view semantic embedding \(\mathbf{s}_{i}^{\ell}\) at each scale \(\ell\): \[c_i^\ell = \frac{\mathbf{s}_i \cdot \mathbf{s}_i^\ell}{\|\mathbf{s}_i\|_2 \, \|\mathbf{s}_i^\ell\|_2}.\]

We also evaluate the reliability distribution of all correspondences by combining the predicted reliability score \(r_i\) from the ground view and \(r_i^\ell\) from the satellite view across all scales \(\ell\): \[w_i^\ell = \frac{\mathrm{exp}(r_i + r_i^\ell)}{\sum_{i',\ell'} \mathrm{exp}(r_{i'} + r_{i'}^{\ell'})}.\] The weights are normalized over all \(N\) nodes and all scales, allowing different correspondences to be emphasized at different resolutions. For example, large objects such as buildings may be matched more reliably at coarse scales, whereas smaller objects such as trees may require finer-scale features.

Finally, the overall matching score for the given pose hypothesis \(\mathbf{H}\) is computed by aggregating the similarity over the whole structure: \[m = \sum_i \sum_\ell w_i^\ell c_i^\ell.\]

Notably, structure-aware matching evaluates each hypothesis based on the holistic spatial consistency rather than isolated point-wise similarities. Under an incorrect hypothesis, a few isolated nodes, such as road regions, may coincidentally project onto visually similar areas and yield high local similarities. However, the overall matching score \(m\) remains low when the majority of the structure is misaligned. This avoids forcing the model to exclusively discriminate semantically consistent objects across viewpoints and instead encourages semantic representations to remain stable under extreme viewpoint changes.

2.4 Hypothesis Ranking↩︎

Structure-aware matching enables CROSS to evaluate a pose hypothesis \(\mathbf{H}\) by its spatial consistency. Instead of optimizing an absolute objective such as direct pose regression, CROSS learns with a relative hypothesis ranking objective. This objective unifies fully and weakly supervised training and encourages transferable geometry for unseen transformations.

2.4.1 Fully Supervised Training↩︎

In the fully supervised setting, where the ground-truth 3-DoF camera pose \((x^*, y^*, \theta^*)\) is available, CROSS learns to rank the ground-truth pose hypothesis \(\mathbf{H}^{*}\) above alternative negative hypotheses.

Specifically, as shown in Fig. 3 (a), we generate \(K\) negative hypotheses \(\{\mathbf{H}_k\}_{k=1}^K\) by applying perturbations to \(\mathbf{H}^{*}\). Structure-aware matching produces a score \(m_k\) for each negative hypothesis \(\mathbf{H}_k\), and \(m^*\) denotes the score of the ground-truth hypothesis. We adopt a Negative Log-Likelihood (NLL) loss [54] to prioritize the ground-truth hypothesis: \[\mathcal{L}_\mathrm{rank} = -\mathrm{log}\left(\frac{\mathrm{exp}(m^*/\tau)}{\mathrm{exp}(m^*/\tau) + \sum_{k=1}^K\mathrm{exp}(m_k/\tau)}\right),\] where \(\tau\) is a temperature hyperparameter.

2.4.2 Weakly Supervised Training↩︎

In the weakly supervised setting, the ground-truth label for the 3-DoF camera pose is unavailable. We consider two weak supervision settings: 2-DoF weak supervision, where the horizontal translation \((x^*,y^*)\) is unavailable while the orientation angle \(\theta^*\) is provided following the setting adopted by most previous weakly supervised cross-view localization methods, and the more challenging 3-DoF weak supervision setting, where accurate labels for both translation and orientation are unavailable.

2-DoF Weak Supervision: When only the orientation label \(\theta^*\) is available, CROSS learns to rank hypotheses with the correct orientation above those with incorrect orientations, while marginalizing over the unknown horizontal translation. Specifically, as shown in Fig. 3 (b), we sample \(K_{\mathrm{ori}}\) negative orientation candidates and generate \(K_{\mathrm{trans}}\) translation hypotheses for each orientation by perturbing the initial translation \((\tilde{x}, \tilde{y})\). Together with the ground-truth orientation \(\theta^*\), this yields \(K_{\mathrm{trans}}\) potential positive hypotheses and \(K_{\mathrm{ori}} \times K_{\mathrm{trans}}\) negative hypotheses. CROSS computes a matching score for each hypothesis. Since the translation label is unavailable, we max-pool the scores over hypotheses sharing the same orientation, producing one score \(m^*\) for the ground-truth orientation and \(K_{\mathrm{ori}}\) scores \(\{m_k\}_{k=1}^{K_{\mathrm{ori}}}\) for negative orientations. The ranking loss is then defined as: \[\mathcal{L}_{\mathrm{rank}} = -\log \left( \frac{\exp(m^*/\tau)}{\exp(m^*/\tau) + \sum_{k=1}^{K_{\mathrm{ori}}}\exp(m_k/\tau)} \right).\]

3-DoF Weak Supervision: In this more challenging setting, where only a noisy initial pose \((\tilde{x}, \tilde{y}, \tilde{\theta})\) is provided, CROSS learns to rank the most plausible pose within an aligned image pair above those from misaligned pairs.

Given a batch of \(B\) aligned ground-view and satellite-view images with noisy initial poses, denoted as \(\{(\mathbf{I}_{i,g}, \mathbf{I}_{i,s}, (\tilde{x}_i, \tilde{y}_i, \tilde{\theta}_i))\}_{i=1}^B\), we perturb each noisy pose \((\tilde{x}_i, \tilde{y}_i, \tilde{\theta}_i)\) to cover the surrounding search region. This yields \(K\) pose hypotheses \(\{\mathbf{H}_{i,k}\}_{k=1}^K\) for pair \(i\).

As illustrated in Fig. 3 (c), CROSS then evaluates matching scores across both positive and negative pairs. For every combination \((\mathbf{I}_{i,g}, \mathbf{I}_{j,s}, \{\mathbf{H}_{j,k}\}_{k=1}^K)\), where \(i=j\) denotes an aligned pair and \(i \neq j\) a misaligned pair, the matching scores of the \(K\) hypotheses are computed. These scores are max-pooled into a single score \(m_{i,j}\), which represents the most plausible pose hypothesis for the image pair \((i,j)\). An NLL loss is then applied to train the model to rank the aligned pair above the misaligned pairs: \[\mathcal{L}_\mathrm{rank} = \sum_{i=1}^B -\mathrm{log}\left(\frac{\mathrm{exp}(m_{i,i} / \tau)}{\sum_{j=1}^B \mathrm{exp}(m_{i,j} / \tau)}\right).\] This allows CROSS to distinguish spatially consistent hypotheses from misaligned alternatives even without precise pose supervision.

2.4.3 Overall Loss↩︎

Although foundation models such as Depth Anything 3 fail to predict reliable and scale-consistent structure across extremely different viewpoints, they still provide strong priors for relative structure. We therefore introduce a monocular-guided loss to distill this prior and refine a scale-consistent structure across viewpoints.

For a ground-view image \(\mathbf{I}_g\), CROSS predicts multi-scale monocular metric-scale depth maps \(\mathbf{D}_g^\ell\) and log-variance maps \(\mathbf{\Sigma}_g^\ell\). Based on the monocular depth map \(\mathbf{D}_g\) predicted by foundation models [8], [55] with arbitrary scale and interpolated to the same resolution as \(\mathbf{D}_g^\ell\), we compute the confidence \(c_i\) and log residual \(g_i\) for each pixel \(i\): \[c_i = \mathrm{exp}(-\sigma_i),~~ g_i = \log(\hat{d}_i) - \log(d_i),\] where \(\sigma_i \in \mathbf{\Sigma}_g^\ell\), \(\hat{d}_i \in \mathbf{D}_g^\ell\), and \(d_i \in \mathbf{D}_g\) represent the predicted log-variance, predicted metric-scale depth and the monocular depth at the \(i\)-th pixel. The global scale shift \(\mu\) is then computed as: \[\mu = \frac{\sum_i c_i g_i}{\sum_i c_i},\] and the monocular-guided loss is defined as the combination of a scale-invariant error and an uncertainty regularization penalty: \[\mathcal{L}_\mathrm{mono} = \sum_i c_i (g_i - \mu)^2 + \sum_i \sigma_i.\] The overall training objective integrates this monocular guidance with the ranking loss: \[\mathcal{L} = \mathcal{L}_\mathrm{rank} + \lambda \mathcal{L}_\mathrm{mono},\] where we empirically set the weighting hyperparameter \(\lambda = 0.1\).

2.4.4 Inference↩︎

During inference, as shown in Fig. 3 (d), given an input ground-view image \(\mathbf{I}_g\), a satellite-view image \(\mathbf{I}_s\), and an initial 3-DoF camera pose \((\tilde{x}, \tilde{y}, \tilde{\theta})\), we first generate pose hypotheses by perturbing the initial pose to cover the surrounding search region. This yields \(K\) candidate pose hypotheses \(\{\mathbf{H}_k\}_{k=1}^K\). CROSS then evaluates the spatial consistency of each hypothesis, computing a corresponding matching score \(m_k\). The final predicted 3-DoF camera pose is derived from the most plausible hypothesis: \[\hat{\mathbf{H}} = \underset{\mathbf{H}_k}{\arg\max} \; m_k, \quad k \in \{1, 2, \dots, K\},\] where the final pose prediction \((\hat{x}, \hat{y}, \hat{\theta})\) is directly extracted from the optimal hypothesis \(\hat{\mathbf{H}}\).

We also conduct coarse-to-fine search which significantly improves inference efficiency as described in Sec. 3.6.

3 Experiments↩︎

In this section, we conduct extensive experiments on two cross-view localization datasets to validate the effectiveness of our method in both localization accuracy and semantic, structural, and geometric learning.

3.1 Datasets, Metrics, and Implementation Details↩︎

3.1.1 Datasets↩︎

We evaluate the proposed method on two widely used cross-view localization benchmarks: KITTI [56] and VIGOR [17]. These datasets cover complementary challenges, including limited ground-view fields of view, large viewpoint gaps, and substantial spatial displacement between ground and satellite views.

KITTI is a standard benchmark collected primarily in urban driving scenarios. The ground-view queries are front-facing camera images, which naturally have a limited field of view (FoV). The corresponding satellite maps cover a \(100~\text{m} \times 100~\text{m}\) region and are centered according to GPS coordinates aligned with the ground-view camera [34]. The dataset is evaluated under two spatial splits: same-area, where training and testing samples are drawn from the same driving trajectories, and cross-area, where testing is performed on geographically disjoint trajectories. Following previous works [18], [19], we conduct experiments with an initial translation noise of \(\pm 20~\text{m}\) and two levels of initial orientation noise, i.e., \(\pm 10^\circ\) and \(\pm 180^\circ\).

VIGOR is a large-scale and realistic cross-view localization benchmark covering four major US cities: New York, Seattle, San Francisco, and Chicago. The ground-view queries are panoramic images, and their true locations are randomly displaced within the central \(1/4\) region of the corresponding satellite-view images. Similar to KITTI, VIGOR provides two evaluation splits: same-area, where training and testing data share geographic regions, and cross-area, where the model is evaluated on unseen cities. Following previous works [18], [19], [28], we evaluate all models under initial orientation noise of \(0^\circ\) and \(\pm180^\circ\) in the fully supervised setting, and \(\pm45^\circ\) and \(\pm180^\circ\) in the weakly supervised setting.

3.1.2 Metrics↩︎

We evaluate cross-view localization performance using the mean and median localization errors in meters, as well as the mean and median orientation errors in degrees.

3.1.3 Implementation Details↩︎

We evaluate our model with both DINOv2 [6] and DINOv3 [7] as frozen backbone encoders. This setting enables a fair comparison with existing baselines, which predominantly adopt DINOv2, while also demonstrating the performance gains brought by a stronger visual foundation model. For spatial representation learning, the DPT decoder outputs feature maps with channel dimension \(C=64\), and the semantic head further reduces the feature dimension to 8. Meanwhile, we use \(D=64\) depth bins ranging from \(1~\text{m}\) to \(100~\text{m}\) for structural learning. For 3D-grounded alignment, we construct a spatial graph with \(N=256\) nodes and initialize the temperature parameter \(\tau\) as 0.07. We use monocular depth maps predicted by Depth Anything 3 [8] and DAP [55] for KITTI and VIGOR, respectively, to provide monocular guidance during training. During training, we generate up to \(K = 17^3 = 4913\) candidate pose hypotheses for ranking. During inference, \(K\) can be further increased to achieve more precise localization. The model is trained for 20 epochs using the AdamW optimizer [57], with a weight decay of 0.01 and a batch size of \(B=8\). The learning rate is linearly warmed up to \(1 \times 10^{-4}\) over the first 500 steps and then decayed with a cosine annealing schedule. All experiments are conducted on a single NVIDIA RTX 6000 Ada Generation GPU.

3.2 Cross-View Localization Comparison↩︎

We compare the proposed CROSS with a broad range of cross-view localization methods under both fully and weakly supervised settings.

3.2.1 Fully Supervised Comparison↩︎

In the fully supervised setting, accurate 3-DoF pose labels are available during training. We compare CROSS with representative methods from different localization paradigms, including global matching methods such as SliceMatch [21] and CCVPE [22], dense matching methods such as HC-Net [24] and DenseFlow [25], and recent sparse matching methods such as FG\(^2\) [18] and Loc\(^2\) [19].

As shown in Tab. 1 and Tab. 2, global matching methods such as SliceMatch and CCVPE generally achieve inferior performance, as they compress the ground-view image into a single global descriptor and inevitably discard fine-grained spatial details. Recent sparse matching methods, including FG\(^2\) and Loc\(^2\), usually outperform dense matching methods such as DenseFlow and HC-Net. This is because sparse matching allows the model to focus on discriminative landmarks for cross-view localization, rather than relying on dense correspondences that can be ambiguous under extreme viewpoint changes. Nevertheless, CROSS consistently achieves stronger performance through 3D-grounded alignment, structure-aware matching, and hypothesis ranking.

On the KITTI dataset, as reported in Tab. 1, CROSS achieves particularly strong performance in the cross-area setting, which better reflects the generalization ability of different methods. Under small orientation noise (\(\pm10^\circ\)), CROSS substantially reduces the localization error compared with previous methods. For example, CROSS-DINOv2 obtains a mean/median localization error of \(3.24/2.00\) m, clearly outperforming Loc\(^2\) with \(5.60/3.01\) m. Under large orientation noise (\(\pm180^\circ\)), the advantage becomes even more significant. CROSS-DINOv2 reduces the mean localization error from \(11.71\) m of Loc\(^2\) to \(3.50\) m, corresponding to an approximately \(70\%\) relative reduction, while also achieving much lower orientation errors. These results demonstrate that CROSS is more robust to severe orientation ambiguity and generalizes more effectively to unseen areas. Moreover, while CROSS-DINOv2 already outperforms most baselines under a fair backbone setting, CROSS-DINOv3 further improves localization accuracy, showing the benefit of stronger visual foundation representations.

In the KITTI same-area setting, CROSS remains competitive, especially under large orientation noise, where it achieves the best orientation accuracy. However, its localization performance is inferior to several baselines in this setting, which should be interpreted with caution. In the same-area split, training and testing frames are sampled from the same driving trajectories and therefore share substantial overlap in scene appearance and spatial layout. Such overlap may allow methods that directly optimize absolute localization targets to overfit to trajectory-specific cues and achieve high accuracy by memorizing nearby frames observed during training. This tendency is supported by their significant performance gap between the same-area and cross-area settings. In contrast, the smaller gap achieved by CROSS suggests that it learns a more transferable localization mechanism, further validating the effectiveness of the proposed hypothesis ranking strategy.

On the VIGOR dataset, as shown in Tab. 2, CROSS consistently outperforms existing baselines across both cross-area and same-area settings. In the cross-area setting with unknown orientation, CROSS-DINOv2 achieves the best localization accuracy with a mean/median error of \(2.25/1.22\) m, substantially improving over previous methods such as Loc\(^2\) (\(4.23/2.09\) m) and CCVPE (\(5.41/1.89\) m). Meanwhile, CROSS also significantly reduces the orientation error, achieving the best mean orientation error of \(5.08^\circ\). When orientation is known, CROSS remains superior, obtaining a mean/median localization error of \(2.00/1.19\) m and outperforming both dense matching methods and sparse matching baselines. Similar trends are observed in the same-area setting, where CROSS achieves the best localization and orientation performance under unknown orientation and remains competitive when orientation is known. These results demonstrate that CROSS enables accurate cross-view localization while maintaining strong robustness to orientation uncertainty and reliable generalization across different geographic regions.

3.2.2 Weakly Supervised Comparison↩︎

Tab. 3 and Tab. 4 report weakly supervised cross-view localization results under noisy pose labels. In the standard 2-DoF weakly supervised setting, accurate position labels \((x^*,y^*)\) are unavailable during training, while the orientation label \(\theta^*\) remains accurate. Existing methods such as G2SWeakly [26] and GeoDistill [28] typically handle translation and orientation separately. On KITTI, the baselines do not explicitly estimate orientation, and their localization performance remains competitive only when the initial orientation noise is small. When the initial orientation noise increases from \(\pm10^\circ\) to \(\pm180^\circ\), their localization errors increase substantially, indicating that inaccurate orientation can severely degrade translation estimation. On VIGOR, the baselines rely on an independent CNN-based orientation estimator, which performs well under moderate initial orientation noise of \(\pm45^\circ\) and achieves a mean orientation error of about \(4^\circ\). However, this estimator breaks down under an initial orientation noise of \(\pm180^\circ\), producing nearly random orientation errors around \(90^\circ\) and causing the localization error to increase from about \(5\) m to about \(15\) m.

In contrast, CROSS does not decouple orientation from translation. Instead, it represents \((x,y,\theta)\) as a joint pose hypothesis and ranks candidate hypotheses according to their cross-view consistency. This coupled formulation allows CROSS to consistently outperform previous methods in the 2-DoF setting, especially under large orientation noise. On KITTI, CROSS reduces the mean localization error under \(\pm180^\circ\) orientation noise to \(7.93\) m and \(7.19\) m on the cross-area and same-area splits, respectively. On VIGOR, the improvement is even more significant. Under \(\pm180^\circ\) orientation noise, CROSS reduces the mean localization error from about \(15\) m to around \(3\) m, while maintaining accurate orientation estimation with mean errors of \(6.94^\circ\) and \(5.62^\circ\) on the cross-area and same-area splits, respectively. These results demonstrate that the hypothesis ranking objective enables the model to jointly reason about translation and rotation, which is crucial for robust weakly supervised localization.

More importantly, CROSS naturally extends to the more challenging 3-DoF weakly supervised setting, where all pose labels \((x^*,y^*,\theta^*)\) are unavailable during training. This setting cannot be directly accommodated by previous weakly supervised methods, as they rely on accurate orientation labels to train a separate orientation estimator. Even under this substantially weaker supervision, CROSS remains robust and still outperforms the 2-DoF weakly supervised baselines across both datasets. For example, under large orientation noise, CROSS achieves mean localization errors of \(11.02\) m and \(9.57\) m on KITTI, and \(4.37\) m and \(4.63\) m on VIGOR, consistently surpassing previous 2-DoF baselines. These results show that the proposed hypothesis ranking objective can simultaneously tolerate noisy translation and orientation labels, making CROSS better suited to realistic weakly supervised scenarios where accurate 3-DoF pose annotations are difficult to obtain.

3.2.3 Visualization of Localization and Matching↩︎

To better understand the localization behavior of CROSS, we visualize cross-view matching and localization results on the KITTI and VIGOR datasets under unknown orientation, as shown in Fig. 4 and Fig. 5. On KITTI, Fig. 4 (a) and Fig. 4 (b) present successful cases with several prominent landmarks, where CROSS establishes reliable correspondences between the ground-view image and the satellite-view image. More importantly, Fig. 4 (c) and Fig. 4 (d) show successful localization in more challenging open-road and natural scenes with fewer salient landmarks. This indicates that the proposed 3D-grounded alignment and structure-aware matching can exploit structural relations among sparse cues, rather than relying solely on local appearance similarities. The failed cases in Fig. 4 (e) and Fig. 4 (f) mainly occur in repetitive road scenes, where the exact position along the road-parallel direction is difficult to disambiguate from visual observations alone. Nevertheless, the predicted orientation remains largely correct, suggesting that CROSS still captures the global directional structure of the scene.

On VIGOR, Fig. 5 (a) and Fig. 5 (b) show accurate localization in urban scenes with clear landmarks, such as buildings and intersections. Fig. 5 (c) and Fig. 5 (d) further demonstrate that CROSS can localize in challenging scenes with fewer distinctive landmarks and repetitive patterns, such as trees, highlighting the importance of structural understanding under ambiguous visual evidence. The failed cases in Fig. 5 (e) and Fig. 5 (f) occur in underground and indoor environments, where the ground-view observations have limited visible correspondences to the satellite map, making accurate cross-view localization nearly impossible.

3.3 Semantic, Structural, and Geometric Learning↩︎

In this subsection, we conduct both qualitative and quantitative analyses to investigate whether CROSS can learn stable semantics, reliable structure, and transferable geometry from cross-view localization.

3.3.1 Stable Semantics↩︎

To investigate the semantic representations learned by CROSS from cross-view localization, we compare the semantic features produced by CROSS and DINOv3. Fig. 6 and Fig. 7 provide qualitative comparisons of cross-view semantic features on the KITTI and VIGOR datasets, respectively. To ensure viewpoint-consistent visualization, we jointly fit PCA [58] on the ground-view and satellite-view feature maps and reduce the feature dimension to three RGB channels. For CROSS, we further use the predicted reliability scores to adjust feature transparency, thereby highlighting reliable landmarks that contribute to localization.

The visualizations show that DINOv3 features exhibit limited cross-view consistency under large viewpoint gaps. Semantically corresponding regions often produce inconsistent color responses between the ground and satellite views, making different areas difficult to align across viewpoints. In contrast, CROSS learns more coherent and viewpoint-consistent semantic representations. Major semantic regions, such as roads, buildings, and trees, are more clearly distinguishable within each view and exhibit more consistent color patterns across the two viewpoints. This demonstrates that cross-view localization provides effective supervision for learning stable semantic representations under extreme viewpoint changes.

We also conduct a quantitative evaluation of unsupervised cross-view semantic segmentation on the VIGOR dataset. We first use Mask2Former [59] and SparseMask [60], [61] to generate segmentation masks for the two viewpoints, covering \(C_s=4\) semantic classes: road, building, tree, and developed space. Then we jointly apply PCA to the pixel-level features from both viewpoints, cluster the reduced features using \(k\)-means, and map each cluster assignment to its corresponding semantic class. As shown in Tab. 5, CROSS substantially improves semantic segmentation performance over DINOv3 in both viewpoints. Specifically, CROSS increases the overall mIoU from 30.10 to 54.55 and the pixel accuracy from 61.39 to 69.40. The improvement is observed for both ground-view and satellite-view imagery, with the corresponding mIoU increasing from 37.25 to 48.58 and from 20.41 to 60.61, respectively. The particularly large improvement of 40.20 percentage points in satellite-view mIoU demonstrates the effectiveness of CROSS in the more challenging satellite-view setting, which contains denser object distributions, more diverse scene elements, and greater visual complexity than ground-view imagery.

We further evaluate whether these semantic representations remain consistent across viewpoints. For each ground-satellite image pair, features belonging to the same semantic class are mean-pooled to obtain one prototype per class for each viewpoint. We then compute a cosine similarity matrix \(\mathbf{S}\in \mathbb{R}^{C_s \times C_s}\) between the ground-view and satellite-view prototypes. Based on this similarity matrix, we report bidirectional Recall@1 (R@1) and the Cross-view Separation Score (CSS), which measures whether semantically corresponding prototypes are more similar across viewpoints than non-corresponding prototypes:

\[\mathrm{CSS} = \frac{1}{C_s} \sum_{i=1}^{C_s} S_{ii} - \frac{1}{C_s(C_s-1)}\sum_{i=1}^{C_s}\sum_{\substack{j=1 \\ j\neq i}}^{C_s} S_{ij}.\]

As shown in Tab. 6, CROSS substantially outperforms DINOv3 across all cross-view semantic consistency metrics. Specifically, CROSS achieves R@1 scores of 76.89 for ground-to-satellite retrieval and 82.21 for satellite-to-ground retrieval, improving over DINOv3 by 22.15 and 26.18 percentage points, respectively. CROSS also increases the CSS score from 0.3952 to 0.7211, indicating that its semantic prototypes exhibit stronger cross-view correspondence while remaining more clearly separated from those of other semantic classes. These results demonstrate that CROSS learns semantic representations that are substantially more discriminative and viewpoint-consistent than those produced by DINOv3.

3.3.2 Reliable Structure↩︎

To further investigate whether CROSS learns reliable and scale-consistent structure from cross-view localization, we visualize its predicted depth maps in Fig. 8. Compared with the binocular depth reference and the directly supervised monocular baseline, CROSS recovers a similar global depth layout, including nearby road regions, intermediate objects such as vehicles and trees, and distant background structures. Although the depth maps predicted by CROSS are smoother and less detailed around thin objects and object boundaries, they still preserve a coherent organization of scene structure and a consistent metric scale without relying on dense metric-scale supervision. This behavior resembles perceptual-level 3D layout understanding, where fine-grained geometric details may be imperfect, but the overall spatial arrangement and scene-scale structure remain consistent.

Table 7 further quantitatively evaluates the metric-scale structure learned on the KITTI dataset, where binocular depth maps predicted by FoundationStereo [62] are used as the reference. Without direct metric-scale depth supervision, CROSS trained with 2-DoF weakly supervised cross-view localization substantially improves over Depth Anything 3 (DA3), reducing AbsRel from \(0.285\) to \(0.235\) and improving \(\delta_1\) from \(0.286\) to \(0.712\). With fully supervised cross-view localization, CROSS further improves all metrics, achieving the best AbsRel, \(\delta_1\), and \(\delta_2\). These results demonstrate that CROSS can refine the scale-consistent structure understanding of current foundation models, such as Depth Anything 3, by learning from cross-view localization, even without direct metric-scale depth supervision.

3.3.3 Transferable Geometry↩︎

To study whether CROSS learns transferable geometry from cross-view localization, we further evaluate the learned model on ground-to-ground matching, where the reference satellite map is replaced by a previous ground-view frame. Since CROSS is trained only on the ground-to-satellite localization task, the key question is whether the geometry learned from this extreme cross-view transformation can transfer to another viewpoint relation without additional supervision.

As shown in Tab. 8, CROSS achieves the best performance on the ground-to-satellite localization task, where it is directly trained. A notable observation is that Depth Anything 3 (DA3), despite showing impressive geometric understanding in ordinary scenarios with moderate viewpoint changes, fails under the extreme viewpoint gap of ground-to-satellite localization, resulting in a mean localization error of \(81.44\) m. In comparison, CROSS effectively learns geometry under large viewpoint changes, leading to robust performance in this challenging cross-view localization setting.

More importantly, CROSS also transfers well to ground-to-ground matching, despite receiving no supervision for this target transformation during training. It achieves a mean localization error of \(1.01\) m and a mean orientation error of \(0.62^\circ\), substantially outperforming Loc\(^2\), which is also trained for ground-to-satellite localization but fails to generalize well to this setting. DA3 obtains the best ground-to-ground performance, benefiting from large-scale training on viewpoint transformations that are closer to this setting. Nevertheless, CROSS achieves competitive results without such supervision, demonstrating that the geometry learned from cross-view consistency is not restricted to the training transformation.

The qualitative results in Fig. 9 further show that CROSS produces coherent ground-to-ground correspondences. For example, buildings and trees are correctly matched across different ground viewpoints under both straight and turning driving conditions. This transferability is mainly enabled by the hypothesis ranking objective. Instead of regressing an absolute pose target, CROSS evaluates candidate pose hypotheses according to their cross-view consistency. As a result, the model is encouraged to learn general geometric reasoning from plausible alignments, rather than memorizing a specific transformation pattern, enabling transfer to unseen viewpoint relations.

3.4 Ablation Study↩︎

In this subsection, we conduct ablation studies to investigate the impact of the proposed modules on both cross-view localization and the learning of stable semantics, reliable structure, and transferable geometry.

3.4.1 Ablation Study on Cross-View Localization↩︎

To evaluate how each module contributes to learning from cross-view localization, we conduct an ablation study on the same-area split of the VIGOR dataset, as shown in Tab. 9. The baseline variant removes all proposed modules and performs direct 2D-2D matching followed by pose estimation, where the model is trained by regressing an absolute pose target. This simple design yields a mean localization error of \(10.70\) m and a mean orientation error of \(51.24^\circ\), indicating that direct 2D-2D matching is insufficient under large viewpoint changes.

Introducing 3D-grounded alignment reduces the mean localization error from \(10.70\) m to \(8.20\) m and the mean orientation error from \(51.24^\circ\) to \(44.85^\circ\). By constructing a 2D-3D-2D pathway, this module encourages the model to reason about reliable scene structure during cross-view localization. The improvement suggests that 3D-grounded alignment provides more reliable candidate correspondences than direct 2D matching for subsequent pose estimation. However, since this variant still mainly relies on point-to-point similarity, its performance remains limited when local appearances are ambiguous or semantically biased.

Adding structure-aware matching further improves the mean localization error to \(4.09\) m and the mean orientation error to \(14.30^\circ\). Instead of enforcing isolated point-to-point correspondences, structure-aware matching aggregates matching evidence over the whole structure. This design helps avoid representation collapse and instance-level bias, allowing semantic features to be learned from consistent structural context rather than individual local matches alone. The substantial performance gain demonstrates that stable semantic learning benefits from evaluating cross-view consistency at the structure level.

Finally, the full CROSS model introduces hypothesis ranking, replacing direct pose regression with a ranking objective over candidate pose hypotheses. This further reduces the mean localization error to \(2.34\) m and the mean orientation error to \(4.28^\circ\), achieving the best performance across all metrics. Unlike regression-based learning from absolute pose targets, hypothesis ranking requires the model to compare the cross-view consistency of different pose hypotheses. This encourages CROSS to learn transferable geometry that is not tied to a fixed transformation pattern, but instead emerges from reasoning over plausible cross-view alignments.

3.4.2 Ablation Study on Semantic, Structural, and Geometric Learning↩︎

We further conduct ablation studies to verify whether the proposed modules, including 3D-grounded alignment, structure-aware matching, and hypothesis ranking, enable the model to learn stable semantics, reliable structure, and transferable geometry.

For semantic learning, as shown in Tab. 10, removing structure-aware matching forces the model to rely on exclusive point-wise matching, where correspondences between semantically consistent but spatially different objects are still heavily penalized. This leads to a substantial degradation in cross-view semantic consistency, with the cross-view semantic retrieval R@1 dropping from around 80 to below 50. The CSS value also decreases, indicating that the learned semantic embeddings become less discriminative across different semantic classes.

For structural learning, we first remove the monocular-guided loss to examine its influence on structure prediction. As shown in Tab. 11, removing only the monocular-guided loss mildly degrades the structure prediction accuracy, while the model can still infer relatively reliable structure through 3D-grounded alignment. We then remove the 3D-grounded alignment module, forcing the model to rely on 2D-2D matching followed by pose estimation. In this setting, the predicted structure is strongly constrained by the 2D matching results, since an incorrect correspondence may require an erroneous depth prediction to recover the correct camera pose. Without explicit 3D grounding, the structure head degrades significantly and fails to predict reliable scene structure, with the AbsRel error increasing from 0.183 to 0.792. This validates that 3D-grounded alignment is the key component for learning reliable structure, while the monocular-guided loss provides a beneficial structural prior.

Finally, we replace the hypothesis ranking objective with an absolute regression objective to evaluate the learned geometry. As shown in Tab. 12, when the model is trained with an absolute objective, it struggles to learn transferable geometry, since it may memorize the specific transformation observed during training rather than evaluating the cross-view consistency of different pose hypotheses. Consequently, its zero-shot ground-to-ground localization performance drops sharply, with the localization error increasing from 1.01 m to 4.09 m and the orientation error increasing from 0.62\(^\circ\) to 27.39\(^\circ\).

Overall, these ablation studies validate the effectiveness of the three key modules in learning stable semantics, reliable structure, and transferable geometry from cross-view localization.

3.5 Robustness Study↩︎

As CROSS supports weakly supervised training with noisy labels, we further investigate its robustness to the initial pose prior in this subsection. In particular, we focus on initial translation noise, since the effect of different initial orientation noise levels has already been reported in Sec. 3.2. As shown in Tab. 13, CROSS remains stable as the initial translation noise increases from \(\pm 10~\text{m}\) to \(\pm 40~\text{m}\). The mean localization error increases only mildly from \(1.91~\text{m}\) to \(2.92~\text{m}\), while the median error remains below \(1.60~\text{m}\) even under the largest perturbation. Meanwhile, the orientation error also changes only slightly, with the median error staying around \(1.2^\circ\)–\(1.3^\circ\). These results suggest that CROSS does not rely on an overly accurate initial pose prior. Instead, it can recover reliable pose estimates from a relatively broad hypothesis region.

3.6 Efficiency Study↩︎

In this subsection, we investigate the efficiency of CROSS. The proposed hypothesis ranking objective enables efficient training, since a small set of negative hypotheses is sufficient to guide the model to distinguish the ground-truth pose from perturbed alternatives. During inference, evaluating more pose hypotheses with finer sampling can improve accuracy, but also increases computational cost. To balance efficiency and effectiveness, we introduce a coarse-to-fine search strategy and compare CROSS with Loc\(^2\) under different input resolutions.

Tab. 14 reports the efficiency results on the same-area split of the VIGOR dataset. The coarse-to-fine search strategy significantly improves the inference efficiency of CROSS with only minor performance degradation. At an input resolution of 1022, it increases the speed from 6.6 FPS to 12.1 FPS, while the mean localization error only changes from \(2.34\) m to \(2.52\) m and the mean orientation error from \(4.28^\circ\) to \(4.96^\circ\). CROSS also shows strong robustness to resolution changes. When the input resolution is reduced from 1022 to 714 and 504, the inference speed further increases to 14.2 FPS and 15.8 FPS, respectively, while the mean localization error only gradually increases from \(2.52\) m to \(2.56\) m and \(2.82\) m, and the mean orientation error from \(4.96^\circ\) to \(5.19^\circ\) and \(6.08^\circ\). Compared with Loc\(^2\), CROSS achieves a more favorable balance between accuracy and efficiency. At the same resolution of 1022, CROSS runs faster than Loc\(^2\), increasing the speed from 8.3 FPS to 12.1 FPS, while substantially reducing the mean localization error from \(7.32\) m to \(2.52\) m and the mean orientation error from \(31.50^\circ\) to \(4.96^\circ\). Notably, even when CROSS uses a much lower resolution of 714 or 504, it still outperforms Loc\(^2\) at a higher resolution of 1428 in both localization and orientation accuracy, while also achieving higher inference speed. These results demonstrate that CROSS is not only more accurate and efficient than Loc\(^2\), but also considerably more robust to input resolution changes.

4 Conclusion↩︎

In this work, we revisit cross-view localization as more than pose estimation and propose CROSS, a unified framework that learns stable semantics, reliable structure, and transferable geometry from cross-view localization. Rather than formulating cross-view localization as 2D-2D matching followed by pose estimation, CROSS introduces 3D-grounded alignment to encourage reliable scene structure learning. It further employs structure-aware matching to overcome the limitations of point-wise matching and learn semantic representations that remain stable across viewpoints. The proposed hypothesis ranking objective supports both fully and weakly supervised training within a unified framework, while promoting geometric understanding that transfers to unseen viewpoint transformations. Extensive experiments demonstrate that CROSS achieves state-of-the-art cross-view localization performance and, more importantly, effectively learns stable semantics, reliable structure, and transferable geometry. In the future, we plan to further explore the potential of CROSS by scaling to larger and more diverse datasets, incorporating broader forms of extreme viewpoint variation, and extending cross-view localization into a general supervision signal for spatial representation learning.

References↩︎

---

1. Mao Chen, Xiangkai Zhang, Zhiyong Liu, and Xu Yang are with the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences.
   Chuankai Liu is with the Beijing Aerospace Control Center.
   (Corresponding author: Xu Yang.)↩︎