1 Introduction↩︎

Global geolocation aims to predict the exact location of any street-view image, with wide applications in security surveillance, emergency response, disease outbreak prediction, environmental monitoring, and tourism navigation [@metageo; @gpt4geo; @geollm]. This task requires integrating visual cues, such as road signs, architectural styles, climate, and vegetation, with geographic knowledge to accurately predict GPS coordinates or location labels. For images with landmarks or distinctive architecture, the location can be inferred by combining visual features with contextual knowledge. However, geolocation becomes more challenging in homogenous environments, such as highways or natural landscapes, where subtle geographic clues like road markings, license plate types, and signage must be relied upon.

Traditional street-view localization methods are generally categorized into retrieval-based and classification-based approaches. The retrieval-based methods [@TransGeo; @cross_view_loc; @zhang2023cross; @GeoCLIP] match input images with similar ones from a geotagged database but are constrained by the diversity and completeness of the database. The classification-based methods [@weyand2016planet; @seo2018; @Translocator] classify images into predefined regions based on visual features, but they lack interpretability and fail to provide explicit visual cues. In practical scenarios, geolocation is rarely a one-time, static process; it involves integrating and refining multiple sources of information iteratively through continuous interaction. Traditional geolocation models inherently lack interpretability and flexibility.

Multimodal Large Language Models (MLLMs) [@llava; @internvl; @llama3] offer a promising alternative due to their ability to synthesize cross-modal information and perform complex interpretive reasoning. This reasoning capacity is the key to unlocking an interactive geolocation paradigm, where spatial ambiguities can be dynamically resolved through multi-turn dialogue, hypothesis generation, and evidence accumulation. Yet, existing general-purpose MLLMs struggle to execute this interactive process, primarily due to a profound geographic knowledge gap [@gpt4geo]. They fail to establish robust, verifiable associations between granular geographic elements (e.g., specific road markings or vegetation patterns) and precise spatial coordinates.

A major bottleneck preventing this paradigm shift is the fundamental misalignment of existing training data. While conventional geolocation datasets, such as OSV-5M [@OSV5M], are engineered exclusively for static, one-shot mapping, general multimodal datasets [@sharegpt; @datacomp] lack the necessary geographic granularity and administrative hierarchies. Driven by the imperative to actualize this interactive geolocation paradigm, we introduce MG-Geo, the first large-scale multimodal dataset explicitly structured to support multi-turn dialogue and deductive spatial reasoning. Featuring an Interactive Reasoning Chain-of-Thought strategy, MG-Geo contains 4.87M geo-tagged Meta entries, 70K image-grounded Clue samples, and 73K multi-turn Dialog samples, with the Clue and Dialog parts organized around eight categories of Geographic Element Cues. As depicted in Figure 1, it covers 210 countries and categorizes cues into eight critical dimensions—including road markings, vegetation, and linguistic signatures. By transitioning from isolated static labels to interconnected reasoning chains, MG-Geo provides the essential foundation for training MLLMs to perform dynamic, evidence-based geographic assistance.

Utilizing this resource, we propose the Global Geolocation Assistant (GaGA), an MLLM architected for high-precision, interactive spatial reasoning. GaGA excels not only in static benchmarks but also in navigating the “Similarity Trap”—a fundamental challenge where geographically widespread visual motifs (e.g., neoclassical architecture) lead to over-confident misclassifications. Our contributions are summarized as follows:

(1) We present MG-Geo, a large-scale multimodal geolocation dataset with 4.87M geo-tagged Meta entries, 70K image-grounded Clue samples, and 73K multi-turn Dialog samples, enabling MLLMs to move beyond static image tagging toward geographic reasoning and interaction.

(2) We develop GaGA, which establishes a new state-of-the-art (SOTA) across multiple benchmarks. On the GWS15k dataset, GaGA surpasses the strong Hybrid baseline by 4.57% and 2.92% at the country and city levels, respectively.

(3) We formalize and investigate the Interactive Geolocation Paradigm. Through a Tiered Interaction Protocol, we demonstrate how GaGA can effectively leverage user-provided anchors to resolve visual ambiguity.

2 Related work↩︎

2.1 Geolocation Datasets↩︎

In the domain of geolocation, the localizability of images within datasets is of paramount importance. Although the existing datasets, such as Im2GPS3k [@vo2017revisiting], YFCC4K [@yfcc100m] and MP-16 [@theiner2022interpretable] contain a large number of geotagged images, many of them are difficult to localize and exhibit distribution biases. GWS15k [@gws15k] mitigates distribution differences, and ensures that the images are authentic, localizable street views. OSV-5M [@OSV5M] is the largest open-source collection of planet-scale, localizable street view images. The Google Landmark V2 [@GoogleLandmark] dataset contains globally distributed human-made and natural landmarks and showcase iconic landscapes.

We introduce MG-Geo, the first large-scale multimodal geolocation dataset comprising 4.87M geo-tagged Meta entries, 70K image-grounded Clue samples, and 73K multi-turn Dialog samples structured with an expert-guided Chain-of-Thought strategy to enhance the geospatial reasoning and interactivity of MLLMs. MG-Geo improves geographic coverage over existing reasoning datasets by combining OSV-5M street-view metadata with GLDv2 landmark imagery, while we visualize its global coordinate distribution and long-tail nature in the Appendix 8. It also incorporates well-structured global language knowledge with eight categories of detailed geographic element cues, providing a dataset that better reflects the complexity and diversity of real-world geolocation challenges.

2.2 Traditional Geolocation Methods↩︎

Traditional mainstream geolocation methods can be broadly categorized into two approaches: image-based retrieval and classification-based methods. Image-to-image retrieval techniques rely on dense image retrieval libraries, which perform well for localization tasks within small areas. However, the cost of constructing such retrieval libraries on a global scale is prohibitively high. When geolocation is treated as a classification task, categories can be defined based on administrative regions, divided into geocells according to specific rules, or discretized into latitude and longitude coordinates. TransLocator [@Translocator] employs images and semantic segmentation maps as inputs, facilitating interaction between two parallel branches after each Transformer layer and enabling multitask geolocation and scene recognition. GeoCLIP [@GeoCLIP] introduces a location encoder and applies random Fourier feature representations to latitude and longitude coordinates. It utilizes the pretrained CLIP [@CLIP] visual encoder to represent images and aligns them with the corresponding location features for localization. PIGEON [@Pigeon] classifies within a self-created geocell and retrieves locations within clusters.

2.3 Generation-Based Geolocation Methods↩︎

Powerful MLLMs have injected new momentum into geolocation research. We refer to this emerging paradigm as generation-based geolocation, which has given rise to two predominant research directions. One dominant approach combines MLLMs with Retrieval-Augmented Generation (RAG) to enhance precision. This line of work includes Img2Loc [@Img2Loc], which reframes geolocation as a training-free text generation problem using CLIP-based retrieval; G3 [@G3] proposes a three-stage framework to mitigate challenges of visual semantics and data imbalance; and GeoRanker [@GeoRanker] introduces a distance-aware ranking framework to model spatial relationships among retrieved candidates. Another research direction focuses on enhancing the intrinsic reasoning abilities of MLLMs. For example, GeoReasoner [@GeoReasoner] fine-tunes an MLLM for street view localization using a curated dataset of human reasoning patterns derived from geolocation games. Similarly, GLOBE [@GLOBE] and GRE [@gre] develop a reasoning-oriented dataset from diverse social media images and use group-relative policy optimization to improve the generation of visual-clue-based rationales. Distinguished from prior work, GaGA is trained on a planet-scale dataset and introduces a novel interactive reasoning paradigm via dialogue.

3 MG-Geo Dataset↩︎

In this paper, we introduce MG-Geo, a novel dataset encompassing a diverse array of geographic element cues, including architecture, environment, landmarks, and climate across various countries. The dataset is structured into three distinct components: the Meta Part, the Clue Part, and the Dialog Part. We leverage structured geographic knowledge, elicited from expert GeoGuessr players, and human-guided interactions with powerful MLLMs to facilitate the construction of this dataset.

3.1 Meta Part↩︎

In the Meta Part, images and meta-geographic information are taken from the OSV-5M [@OSV5M], which inherits its characteristics of good distribution, wide scope, and high quality. After removing a small number of samples with incomplete location annotations, we organize each sample into JSON format using three levels of administrative boundaries—country, region, and city. This results in a total of 4.87 million entries, covering 70k cities, 2.7k regions, and 210 countries.

3.2 Clue Part↩︎

To enhance interpretability, we develop an automated multimodal QA generation paradigm that transforms raw geographic annotations into structured image-text clue pairs. Figure 2 (a) illustrates this high-correlation data generation pipeline.

MLLM-Based Refinement. While the 3,000 expert clues from GeoGuessr and Tuxun provide rich localization knowledge, pure text lacks the visual context necessary for effective multimodal reasoning. To address this, we leverage MLLMs to align each clue with its corresponding visual features. We sampled 70k globally distributed images from OSV-5M [@OSV5M] and implemented a two-step process: (1) constructing country-specific clue repositories via manual classification, and (2) automated image-clue matching to associate street-view imagery with relevant regional descriptors.

Human Verification. Despite rigorous automated matching, visual ambiguities (e.g., low resolution or missing views) may persist. We implemented a manual validation protocol to identify and rectify erroneous pairs. Evaluators trace error sources to either prune problematic samples or recalibrate metadata, ensuring high fidelity between natural language descriptions and their visual targets.

3.3 Dialog Part↩︎

As shown in Figure 2 (b), we begin the Dialog Part construction process by standardizing a well-annotated subset of the Google Landmark V2 into a unified metadata structure, ensuring the generation of multi-turn reasoning QA pairs that are location-agnostic. In order to enhance GaGA’s reasoning depth and conversational ability by supporting the analysis of images from multiple perspectives and inferring specific locations, we select 73K samples from Google Landmark V2 with rich information such as architecture, vegetation, cultural elements, and climate. Then, with the assistance of GPT-4V, we generate QA pairs using the Interactive Reasoning CoT method.

Question-Answer Generation We intend to create image descriptions that thoroughly capture visible appearance and attributes, integrating relevant knowledge, climatic characteristics, architectural styles, and even historical context. This all-encompassing strategy ensures the dataset’s robust support for a broad spectrum of real-world applications by providing enriched and contextually rich data.

The generation of multi-turn QAs mainly relies on providing unified metadata and carefully designed prompts to MLLMs, specifically GPT-4V. Through this process, GPT-4V engages in multi-turn self-questioning based on the image, gradually guiding the model to reason through and uncover the geographical information. Each set of multi-turn QAs includes the following key attributes: question ID, source dataset, image path, three geo-questions w/ reasoning process, and one loc-question w/ ultimate answer. This structure ensures the logical coherence of the multi-turn QAs and clearly presents the progression from question to reasoning process to the final answer.

Prompting techniques improve LLMs’ reasoning and problem solving abilities across diverse tasks  [@zero-shot-planners; @Swe-bench; @Vipergpt]. We integrate the images’ unified metadata format to generate high-quality dialog data. Using the Interactive Reasoning CoT method, we activate two tasks: CoT Deduction and CoT Introspection. In the next part, we elaborate on the implementation details of these two tasks.

CoT Deduction. To extract the reasoning chain behind the geographic location predictions from GPT-4V as the training data, we explicitly extract the reasoning chain supporting the model’s QA process. Specifically, we draw on the concept of interactive reasoning from reinforcement learning and propose the CoT Deduction method to handle the geographic location prediction task.

The CoT Deduction consists of three parts: (1) Role Setting. In CoT Deduction, we set up two roles: Geo-Guessr player and questioner. The questioner and player interact, with the questioner asking questions and the player responding based on the image clues and existing knowledge. (2) Reasoning QA. We aim to explicitly extract the internal principles of geographic location reasoning to construct MG-Geo’s Dialog Part. For each question from the questioner, the Geo-Guessr player gradually deduces the geographic location based on various aspects embedded in the image, such as the environment and climate, architecture and landmarks, language and culture, and people’s appearance. Each QA round (i.e., Q1A1, Q2A2, and Q3A3) helps the player narrow down the possibilities, gradually approaching the correct answer. (3) Coordinate Prediction. After a series of reasoning steps, the player needs to provide a specific geographic coordinate and briefly explain their choice (Q4A4).

After CoT Deduction generates the predicted coordinates, we initiate a Decision Criterion to evaluate the predicted coordinates’ accuracy. Specifically, we calculate the Haversine distance between the predicted coordinates and the unified metadata. If the distance between the predicted and true coordinates is greater than \(25km\), the CoT Introspection process is triggered.

CoT Introspection. If the initial coordinate prediction deviates from the metadata by more than 25km, we trigger CoT Introspection. Instead of treating the ground-truth location as a shortcut answer, this stage asks the MLLM to re-check whether each reasoning step is visually grounded, whether the selected clues are geographically discriminative, and whether any high-specificity evidence has been overlooked. The resulting sample is retained only when the revised reasoning remains supported by visible image evidence and passes the quality-control criteria. We explicitly tag all such samples as introspection; in MG-Geo, 70.1% of the Dialog Part, approximately 51K/73K samples, follows this path. On the triggered subset, the original pre-refinement deduction already reaches 85.2% Acc@750km and 60.2% Acc@200km, suggesting that Introspection mainly refines directionally correct but insufficiently fine-grained reasoning rather than replacing arbitrary predictions.

4 The GaGA Model↩︎

Capitalizing on the MG-Geo dataset, we present GaGA, an interactive MLLM designed to transcend the "black box" nature of traditional geolocation. As illustrated in Figure 3, GaGA shifts the geolocation paradigm from static coordinate regression to a dynamic reasoning process. By integrating robust spatial awareness with extensive world knowledge, GaGA can effectively leverage logical constraints during user interaction to refine its predictions in real-time.

4.1 Model Architecture↩︎

GaGA adopts a modular MLLM architecture following the LLaVA paradigm [@llava], optimized for iterative spatial reasoning. The framework integrates a vision encoder \(f_{V}\), a cross-modal projector \(f_P\), and an LLM \(f_L\) (Llama3-8B [@llama3]) as the central reasoning engine.

Visual Encoding and Projection. For an input street-view image \(X_v\), we employ a pretrained CLIP-ViT [@vit] as the vision encoder \(f_{V}\) to extract high-level semantic features. To bridge the modality gap, the projector \(f_P\) (a two-layer MLP) maps these visual representations into the LLM’s latent word embedding space. The resulting visual tokens are obtained via \(E_v = f_{P}(f_{V}(X_v))\).

Multimodal Contextualization and Reasoning. To support the interactive geolocation paradigm, the text tokenizer \(f_T\) is designed to process not just a single query, but a progressive reasoning trajectory. It transforms the cumulative dialogue history \(H_{<n}\) and the current user query \(Q_n\) (which may contain high-specificity geographic priors, such as language or traffic rules) into textual tokens \(E_{T,n} = f_T([H_{<n}; Q_n])\). These tokens are concatenated with \(E_v\) to form a unified input sequence. The LLM then performs auto-regressive inference to generate a refined reasoning trajectory \(R_n = f_{L}([E_v; E_{T,n}])\). This architecture facilitates a deep cross-modal exchange, allowing \(f_L\) to contextualize abstract visual patterns within the provided geographic constraints, sequentially updating its belief state to escape visual ambiguities.

4.2 Training Strategy and Objectives↩︎

The training protocol of GaGA is conceptualized as a two-stage paradigm: Geographic Alignment and Interactive Instruction Tuning. This progressive approach ensures that the model first internalizes a global geographic world-view before mastering complex spatial reasoning and dialogue capabilities.

Stage 1: Geographic Alignment. The primary objective of this stage is to bridge the modality gap between generic visual features and structured geographic metadata. We initialize the multimodal projector with ShareGPT4V [@sharegpt] weights to leverage its robust cross-modal mapping. During this phase, both the vision encoder and the LLM backbone are kept frozen to preserve their respective pre-trained representational power.

We optimize the projector \(\phi\) by minimizing the standard cross-entropy loss over the Meta Part of the MG-Geo dataset. Formally, given an image \(X_v\) and its associated geographic metadata \(Y_{geo}\) (comprising country, region, city, and coordinates), the training objective is: \[\mathcal{L}_{align} = -\sum_{t=1}^{T} \log P(y_t | X_v, y_{<t}; \phi),\] where \(Y_{geo} = \{y_1, ..., y_T\}\) represents tokenized geographic attributes. This stage forces the projector to learn a Spatial-Aware Embedding that aligns visual patterns with specific global hierarchies.

Stage 2: Interactive Reasoning Refinement. The second stage focuses on cultivating Constraint-based Reasoning and multi-turn adaptability. We freeze the projector \(\phi\) and apply QLoRA [@QLoRA] to the LLM to optimize its decision-making logic. The training corpus integrates the Meta, Clue, and Dialog parts of MG-Geo, with a strategic emphasis on the Interactive Reasoning CoT strategy. Crucially, the training corpus here is a heterogeneous mixture of 240k carefully curated samples from the Meta, Clue, and Dialog parts of MG-Geo. The Clue Part explicitly exposes the model to 8 distinct dimensions of geographic evidence (e.g., architecture, vegetation, road markings), while the Dialog Part instills the Interactive Reasoning CoT strategy. The optimization goal is to maximize the conditional likelihood of the correct reasoning chains \(R\) and the final coordinates \(C\) given the visual input \(X_v\) and user queries \(Q\): \[\mathcal{L}_{ft} = -\mathbb{E}_{(X_v, Q, R, C) \sim \mathcal{D}_{MG-Geo}} [ \log P(R, C | X_v, Q; \theta_{LoRA}) ].\]

By tightly coupling the LLM optimization with MG-Geo’s conversational structure, this multi-task objective enables GaGA to identify and rectify its initial misjudgments. This ensures that the final response is the result of a deliberate, multi-turn evidentiary synthesis rather than a simplistic statistical mapping.

5 Experiment↩︎

To demonstrate the efficacy of our dataset in addressing the geographic knowledge gap in existing models and to showcase GaGA’s ability in geolocation tasks, we conducted a comprehensive suite of experiments, the primary findings of which are presented in this section.

5.1 Experimental Setup↩︎

Metrics. We use the following metrics to evaluate the prediction accuracy of the geolocation model:

(1) Accuracy of predicted geographical names at various administrative levels: country, region and city.

(2) Accuracy of predicted coordinates within various distance thresholds: 1km, 25km, 200km, 750km, and 2500km, calculated as the haversine distance between the model’s predicted GPS coordinates and the ground truth.

(3) Geoscore: it is defined as \(5000exp(-\delta/1492.7)\) based on the famous “Geoguessr” game. \(\delta\) represents the Haversine distance between predicted and ground truth image locations.

Benchmark. We assess GaGA’s performance across multiple benchmarks. For its primary global geolocation capability, we use GWS15k [@gws15k] and OSV-5M-test [@OSV5M] for its balanced worldwide coverage. Since GWS15k is not open-source, we reproduce it in this study. The pseudocode and data distribution are presented in the Appendix 11. To further evaluate its generalization capability, we conduct comparative experiments between GaGA and state-of-the-art methods on two web-crawled image benchmarks, Im2GPS3k [@vo2017revisiting] and YFCC4k [@yfcc100m], with results reported in the Appendix 12.

Implementation Details. All experiments are conducted using the XTuner platform [@xtuner], facilitating efficient multimodal model tuning and deployment. For reasoning tasks, we use LMDeploy [@lmdeploy] tool. We conduct all the experiments on 8 \(\times\) RTX4090 GPUs.

5.2 sec:Geolocation32Performance↩︎

The evaluation results on the GWS15k benchmark are summarized in Table ¿tbl:tab:32Geolocation32performace?, where we categorize current methodologies into traditional discriminative models and MLLM-based reasoning assistants.

As shown in Table ¿tbl:tab:32Geolocation32performace? (Left), GaGA achieves outstanding performance in administrative boundary identification. Notably, while general-purpose MLLMs (e.g., LLaVA-Llama3, Qwen-VL [@Qwen2.5-VL; @Qwen3-VL]) and even specialized geographic MLLMs (e.g., GeoReasoner [@GeoReasoner], GLOBE [@GLOBE]) struggle with precise localization, GaGA achieves SOTA performance among all MLLM-based methods. Specifically, GaGA attains a country-level accuracy of 63.06% and a city-level accuracy of 6.28%, outperforming the previous best-performing MLLM, GLOBE, by a substantial margin of 15.97% and 2.84% respectively. Even when compared to traditional SOTA classifiers like Hybrid [@OSV5M], GaGA maintains a clear lead (+4.57% at country level and +2.92% at city level), demonstrating that the interactive reasoning paradigm does not sacrifice, but rather enhances, fundamental localization precision. Furthermore, GaGA maintains a 100% Recall rate, ensuring that every query is met with a valid geographic coordinate—a critical reliability metric where other LLMs often falter.

Table ¿tbl:tab:32Geolocation32performace? (Right) presents the coordinate-level error distribution. Among MLLM-based approaches (highlighted in the bottom section), GaGA consistently defines the state-of-the-art, significantly surpassing GeoReasoner, GRE, and GLOBE across thresholds of 25 to 2500km. For instance, at the 200km threshold—a key metric for regional localization—GaGA leads the second-best MLLM (GRE) by 15.0%. When compared to traditional end-to-end models, GaGA exhibits competitive performance, securing the second-best results from 200km to 2500km. Most importantly, GaGA achieves the highest Geoscore (3113.0) among all reproducible models, including the strong Hybrid baseline. This peak Geoscore underscores GaGA’s unique ability to strike an optimal balance between high-precision hits and the mitigation of catastrophic outliers, a direct result of its clue-based reasoning and interactive refinement capabilities.

Table [tbl:tab:comparison95on95osv5m] reports the comparative results on the OSV-5M test set. Notably, GaGA achieves the highest City-level accuracy (7.4%) among all competing methods. In terms of global metrics, GaGA reaches a Geoscore of 3413 and a Country-level accuracy of 71.4%, consistently outperforming specialized models like ISNs and the Hybrid baseline. These findings underscore GaGA’s robustness in balancing macro-scale region recognition and micro-scale city identification.

5.3 Inference-Time Interactive Geolocation Analysis↩︎

A fundamental strength of GaGA lies in its capacity for Multi-tier Interactive Reasoning, allowing it to dynamically refine geolocation beliefs through dialogue. In this section, we provide a formal dissection of this capability. First, we establish a taxonomic framework for interactive guidance to quantify performance gains across different levels of information density. Second, we investigate the "Similarity Trap" to elucidate how the specificity of geographic cues influences the model’s posterior distribution.

5.3.1 Effectiveness of Tiered Interactive Guidance↩︎

To evaluate GaGA’s reasoning flexibility, we curated a diagnostic set of 547 images spanning diverse cultural and natural landscapes. We standardized the interaction into three progressive tiers: (1) Zero-shot (Direct Inquiry): Baseline prediction without external cues. (2) Tier-1 (Focus-based Inquiry): Providing a guiding question (e.g., “Considering the architectural design, what region of the world would you think displays such forms?”) to direct the model’s attention to specific attributes without providing factual answers. (3) Tier-2 (Knowledge-augmented Guidance): Providing both a question and a factual geographic prior (e.g., “The photo shows the typical oceanic and continental climate zones; which countries typically use this color?”).

Table ¿tbl:tab:geographic32knowledge32question32answering? illustrates the performance dynamics. The results reveal several key insights. Firstly, Tier-2 interaction yields the most significant gains, with GaGA’s country-level accuracy surging by 9.88% (reaching 74.77%). This demonstrates GaGA’s superior ability in Knowledge Integration—effectively using external anchors to prune the search space. Secondly, an intriguing observation arises in Tier-1 interaction: while region and city-level precisions improve, country-level accuracy experiences a slight decline (-3.65%). We characterize this as the exploration cost within the Similarity Trap. Without a factual anchor, Tier-1 queries compel the model to focus on broadly distributed visual features. While this sharpens the model’s focus on local details (improving city-level hits), it simultaneously increases the risk of being misled by cross-border visual similarities.

5.3.2 Geographic Ambiguity and the “Similarity Trap”↩︎

Global geolocation is inherently plagued by visual ambiguity, a challenge we formalize as the “Similarity Trap”. This occurs when geographically widespread features—such as neoclassical architecture or generic temperate flora—produce overlapping representations in the model’s feature space, leading to over-confident misclassifications [@around_the_world_in_80_timesteps]. While static retrieval-based models are often paralyzed by such ambiguity, GaGA’s interactive framework introduces a dynamic Error-Correction Mechanism.

As illustrated in the comparative analysis in Figure 4, when GaGA is prompted with an Unconstrained Semantic Heuristic (e.g., “How does the background greenery help identify the location?”), it may initially succumb to the trap by over-relying on generic motifs that lack geographic exclusivity. However, by transitioning to a High-Specificity Structural Constraint (e.g., “How do the oceanic and continental climate features aid in localization?”), the model is forced to decouple ambiguous visual signals and engage in constraint-based reasoning. This logical anchoring allows GaGA to suppress noise and re-route its reasoning toward the correct manifold.

Formally, let \(G\) denote the geographic label and \(c\) a visual clue. We define clue specificity as \[s(c) = 1 - \frac{H(G|c)}{H(G)},\] where low-specificity clues have high conditional entropy over possible locations, while high-specificity clues sharply reduce the uncertainty. Given model salience weights \(\alpha_i\) over visual clues \(\{c_i\}\), a Similarity Trap occurs when the prediction is dominated by low-specificity clues, i.e., \(\sum_i \alpha_i(1-s(c_i))\) is high, leading to a localization error \(d(\hat{y}, y^*)>\tau\).

Based on this specificity criterion, we partitioned our evaluation set into two diagnostic subsets based on the information entropy and geographic exclusivity of the cues:

(1) High-Specificity Subset: Focuses on Deterministic Markers with narrow geographic distributions and strong logical constraints (e.g., road markings, signage language, or driving side).

(2) Ambiguous Subset: Focuses on Distributive Features with high spatial variance and weak exclusionary power (e.g., architectural styles, general vegetation, or terrain).

The empirical results in Table [tbl:tab:ambiguity95impact] confirm that high-specificity cues yield significantly higher precision across all administrative levels. This validates our hypothesis that the “Similarity Trap” is the primary bottleneck in geolocation. More importantly, it underscores the necessity of GaGA’s interactive paradigm: by converting human-provided high-specificity priors into logical constraints, GaGA effectively navigates through visual ambiguity where static models fail.

5.4 Ablation Experiments↩︎

To systematically evaluate the contribution of each component within the MG-Geo dataset, we conduct a comprehensive ablation study. Using the test set from Section 5.3.1, we design five experimental groups. We evaluate performance across four key metrics: Accuracy (Geo-Score), Informativeness (hit rate of generated clues, assessed by an LLM Judge), Fluency (Likert scale score for coherence, assessed by an LLM Judge), and Relevance (contextual relevance, assessed by an MLLM Judge).

As detailed in Table ¿tbl:tab:ablation?, our ablation study validates the indispensable roles and intentional design trade-offs of the MG-Geo dataset components. The Meta data serves as the bedrock for static localization, achieving the highest overall Accuracy (3872). Conversely, the Dialog component acts as the primary driver for conversational reasoning, yielding substantial improvements across interactive metrics, including a significant leap in Informativeness (from 0.6164 to 0.7549). Furthermore, the Clue data functions as a conditional enhancer; while it can slightly perturb accuracy when lacking conversational context, combining it with the reasoning framework (Meta+Clue+Dialog) maximizes both Informativeness (0.7809) and Fluency (4.9555). Ultimately, this deliberate trade-off—sacrificing a marginal degree of raw precision for massive gains in explainability and interaction—confirms our core objective of developing a balanced, interactive global geolocation assistant.

We further isolate the effect of CoT Introspection by removing all Introspection-tagged dialogs while maintaining the same Stage-2 sample budget. The full model improves over the w/o-Introspection variant by +5.74 Acc@200km, +12.77 Acc@750km, +15.00 Acc@2500km, and +7.23 country-level accuracy, supporting the role of Introspection as a cue-correction signal rather than a coordinate shortcut. Detailed results are provided in the Appendix 7.

5.5 Release, Privacy, and Responsible Use↩︎

MG-Geo is released in a source-compatible form, including annotations, prompts, splits, generation-path tags, and reconstruction scripts.

6 Conclusion↩︎

In this work, we address key challenges in global geolocation by tackling the critical lack of comprehensive data and the performance limitations of existing methods. To this end, we introduce MG-Geo, the first large-scale, high-quality multimodal dataset specifically designed with rich geographic cues to bridge this knowledge gap for MLLMs. Building upon this foundational dataset, we developed GaGA, a novel MLLM that not only demonstrates superior performance over existing models in predicting administrative boundaries but also introduces a crucial interactive capability for refining localization through user dialogue. Our work underscores the critical role of high-quality, domain-specific datasets in unlocking the potential of MLLMs for complex spatial reasoning tasks and opens new avenues for a wide range of geographic downstream applications.

Acknowledgements↩︎

This work was supported in part by the Key Deployment Program of the Chinese Academy of Sciences, China under Grant KGFZD-145-25-39, the National Natural Science Foundation of China under Grants 62272438, and Beijing Natural Science Foundation L25700.

Appendix↩︎

7 CoT Introspection Analysis↩︎

7.0.0.1 Trigger and tagging.

After CoT Deduction predicts coordinates, we compute the Haversine distance between the prediction and the metadata coordinates. CoT Introspection is triggered when this distance exceeds \(25\,\mathrm{km}\). In MG-Geo, 70.1% of the Dialog Part, approximately 51K out of 73K samples, follows this path. We explicitly tag every Dialog sample with its generation path, either deduction or introspection, so downstream users can include, exclude, or separately analyze Introspection-generated samples.

7.0.0.2 Role of Introspection.

Simply feeding a ground-truth answer back to an MLLM can lead to post-hoc rationalization if left uncontrolled. Our intended use of Introspection is therefore evidence re-checking rather than answer-forced explanation. Given a verified geographic anchor, the model is asked to re-examine whether each reasoning step is visually grounded, whether the selected clues are geographically discriminative, and whether high-specificity evidence has been overlooked.

On the triggered subset, the original pre-refinement deduction already reaches 85.2% Acc@750km and 60.2% Acc@200km. This indicates that Introspection mainly refines directionally plausible but insufficiently fine-grained reasoning, rather than replacing arbitrary predictions with a known answer. No ground-truth coordinates are provided during downstream model evaluation.

The ablation removes all Introspection-tagged dialogs while maintaining the same Stage-2 sample budget by resampling non-Introspection Clue/metadata samples under the identical training recipe. The full model improves over the w/o-Introspection variant most strongly at coarse-to-mid coordinate thresholds, suggesting that Introspection functions as a cue-correction signal that reduces catastrophic far-away predictions.

8 Geographic Distribution and Regional Performance↩︎

MG-Geo provides broad geographic coverage but naturally follows a long-tailed distribution inherited from global street-view and landmark availability. To make this distribution transparent, we report both sample density and regional error patterns. Fig. 5 visualizes the global coordinate distribution of MG-Geo, while Fig. 6 shows the geodesic error distribution of GaGA on the reproduced GWS15k-style benchmark.

We release the full per-country distribution as a CSV file together with the dataset card. MG-Geo is characterized as a broad-coverage but long-tailed resource, and this distribution is exposed for downstream analysis.

9 Backbone Transferability↩︎

To verify that the gains of MG-Geo are not tied to a single LLaVA-Llama3 backbone, we fine-tune Qwen3-VL-8B with the same MG-Geo supervision recipe and evaluate it on the same reproduced GWS15k-style benchmark. Table 3 shows that MG-Geo supervision provides large gains across both coordinate-level and administrative-level metrics.

For Qwen3-VL-8B, Stage 1 trains only the multimodal projector on the Meta Part. Stage 2 starts from the Stage-1 checkpoint, freezes the vision encoder and multimodal projector, and applies LoRA to the language model using the Dialog Part. This mirrors the two-stage MG-Geo training protocol and isolates the contribution of the dataset from the original GaGA backbone.

10 Release Plan, License Compatibility, and Responsible Use↩︎

10.0.0.1 Release scope.

We release MG-Geo in a source-compatible form, including annotations, train/validation/test splits, prompts, generation-path tags, and reconstruction scripts. Raw images remain governed by the licenses and access terms of their original sources.

10.0.0.2 Generation-path transparency.

All Dialog samples are tagged with their generation path, either deduction or introspection. This allows downstream users to filter the dataset according to their tolerance for Introspection-generated reasoning chains.

10.0.0.3 License compatibility.

MG-Geo combines geo-tagged street-view metadata and landmark imagery from existing sources. We do not re-license raw images. Instead, we provide source identifiers and reconstruction scripts where appropriate. AI-generated annotations are explicitly marked and released subject to applicable terms of the data sources and model providers.

10.0.0.4 Privacy and dual-use.

Street-view imagery may contain sensitive visual content such as faces, license plates, private property, or location traces. MG-Geo is intended for research on geolocation reasoning, benchmark evaluation, and spatial understanding. It should not be used for identifying individuals, tracking private persons, targeted surveillance, or other privacy-invasive applications. We recommend downstream users follow the privacy handling and removal mechanisms of the original image sources.

11 Reproduction of the GWS15k-style Benchmark↩︎

Because the original GWS15k split is not publicly released, we construct a GWS15k-style benchmark from OSV-5M-Test for protocol-level comparison. Results on this reproduced split are marked with \(^\bigstar\) in the main paper and should not be interpreted as exact split-level comparisons to prior GWS15k reports.

We use the test set of OSV-5M as the database and collect geographically distributed imagery based on 43K cities and the surface area of each country. We first sample countries/regions based on their proportion of Earth’s surface area, then randomly select a city within each and GPS coordinates within a 5 km radius of that city’s center to sample from OSV-5M-Test. The pseudocode is presented in Algorithm 8, and the sampled data distribution is illustrated in Fig. 7.

12 Additional Standard Benchmarks↩︎

When evaluating generalization on Im2GPS3k and YFCC4k, it is important to acknowledge their data distribution limitations. A substantial proportion of images within these datasets lack discernible geographic cues, so performance metrics can be influenced by dataset-specific biases rather than only by spatial reasoning ability [@OSV5M].

Because Im2GPS3k and YFCC4k contain many non-street-view and retrieval-dependent cases, we additionally report a retrieval-augmented variant, GaGA (MP16-Pro), for reference. This version equips GaGA with a Retrieval-Augmented Generation module that uses the large-scale MP16-Pro dataset as an external knowledge base. These results are intended to show compatibility with external retrieval resources rather than to replace the main street-view geolocation evaluation.

The performance comparisons on Im2GPS3k and YFCC4k are presented in Table 4. We categorize the baselines into Traditional Methods and Generation-based Methods. GaGA (MP16-Pro) improves substantially over the base GaGA model, especially on retrieval-dependent cases. This suggests that a reasoning-focused model can serve as a strong foundation when combined with external retrieval tools.

3pt

13 Hyper-parameter Settings↩︎

Tables ¿tbl:tab:pretrain-settings? and [tbl:tab:finetune-settings] summarize the hyper-parameters used in the two training stages. Parameters not mentioned in Stage 2 are the same as those in Stage 1.

14 Prompts Employed in MG-Geo Dataset Generation↩︎

To ensure question variety, we design multiple templates for each question type. These templates provide variation while maintaining focus on the geolocation task. The prompts below show the instruction templates used for clue extraction and dialogue generation.

The CoT Deduction prompt guides the model through reasoning and prediction, while the CoT Introspection prompt asks the model to re-check evidence under a verified geographic anchor.

15 Cue Distribution in the Clue and Dialog Parts↩︎

We further analyze the cue distribution in the Clue and Dialog Parts. For the top-20 countries by sample frequency, we extract the four most frequent direct cue phrases. As shown in Fig. 9, the Clue Part covers more fine-grained geographic evidence, while the Dialog Part emphasizes conversational reasoning around buildings, environment, architecture, and climate. Across the eight cue categories, architectural features, vegetation, and road styles are the most frequent evidence types in street-view geolocation.

16 Performances of Advanced MLLMs in Dialog↩︎

Table ¿tbl:tab:advanced95vlm? illustrates the performance of various MLLMs under different interaction paradigms, comparing Direct Inquiry against Tier-1 Focus-based Inquiry. Comparing GaGA with its base model, LLaVA-Llama3, we observe that under Focus-based Inquiry, GaGA achieves precision gains at the region and city levels. The country-level decline under Tier-1 focus-based inquiry also reveals a limitation: non-discriminative focus questions can make the model over-attend to weak visual motifs, causing brittle reasoning under ambiguous cues. Tier-2 factual anchors mitigate this issue by converting broad visual attention into testable geographic constraints.

Furthermore, we evaluate the impact of this interactive design on other general-purpose MLLMs, including InternVL2 [@Internvl2] and Qwen-VL [@Qwen-VL]. While Qwen-VL struggles under the standard Direct Inquiry setting, its country-level accuracy improves significantly when prompted with a Focus-based Inquiry. Similarly, through guided dialogue, InternVL2 uncovers actionable visual clues and improves across administrative levels. These results suggest that interactive reasoning can activate latent geographic knowledge and mitigate visual ambiguity across different MLLM architectures.

We employ two evaluation modes, hierarchical (HIER) and direct (DIRE). HIER is primarily used for MLLMs that have not been finetuned on MG-Geo; in this mode, candidate administrative boundary names are provided at each level to constrain the output space.

17 Ablation on Training Stages↩︎

As shown in Table ¿tbl:tab:model95comparison?, we evaluate the impact of the two training stages. GaGA-Stage1 achieves slightly higher static administrative-level accuracy, while GaGA-Stage2 introduces dialog-based reasoning and improves the model’s ability to integrate user-provided constraints. This trade-off is consistent with the goal of building an interactive geolocation assistant rather than a purely one-shot classifier.

18 Additional Qualitative Results↩︎

To further illustrate GaGA’s localization behavior across diverse global scenarios, we present additional qualitative results in Fig. 10.