TerraLogic: A Benchmark for Hierarchical Geospatial Reasoning in Earth Observation
July 14, 2026
Abstract
Beyond perception, reasoning is essential in remote sensing for advanced interpretation, inference, and decision-making. Recent advances in large language models (LLMs) have enabled tool-augmented agents that leverage external tools to perform complex analytical tasks. However, existing studies in remote sensing primarily focus on perception-oriented tasks, leaving cognitive geospatial reasoning largely underexplored. To address this gap, we introduce TerraLogic, a benchmark for geospatial reasoning. TerraLogic comprises 545 scenario-driven, hierarchy-aware tasks—such as hazard vulnerability assessment, urban heat island analysis, and forest fragmentation dynamics—spanning optical, Synthetic Aperture Radar (SAR), and infrared (IR) imagery. It advances evaluation beyond recognition and monitoring toward cognitive-level geospatial analysis. To facilitate evaluation on TerraLogic, we further propose HieraPlan, a tool-augmented agent that organizes toolkits into functional hierarchies and performs fault-tolerant reasoning. HieraPlan enables structured abstraction, robust recovery from tool failures, and stable long-horizon planning. Extensive experiments demonstrate that current approaches struggle with hierarchical geospatial reasoning, while HieraPlan provides a strong baseline with improved reasoning, cross-modal generalization, and error handling. The dataset and agent code are publicly available at https://github.com/Ireliya/TerraLogic.
Keywords
Benchmark dataset; Geospatial reasoning; Intelligent agents; Remote sensing; Multimodal reasoning; Tool-augmented agents.
1 Introduction↩︎
Large language models (LLMs), empowered by generative pretraining and instruction tuning, have substantially improved zero-shot task completion across diverse applications [1], [2]. Building on this progress, LLM-driven agents can decompose high-level goals into sub-tasks and orchestrate external tools for multi-step problem solving [3], [4]. However, extending such agents to remote sensing (RS) remains challenging. RS imagery is highly heterogeneous across modalities such as optical, Synthetic Aperture Radar (SAR), and infrared, while targets vary drastically in scale, orientation, and spatial configuration. These properties often challenge the assumptions of generic visual-language reasoning, causing agent performance to degrade under RS-specific data distributions and stricter constraints on spatial units and tool parameters.
More importantly, real-world Earth observation (EO) applications require more than perception. Beyond perception-oriented tasks such as classification, localization, counting, and RS Visual Question Answering (VQA), many EO applications demand cognitive-level geospatial reasoning for decision-making. These tasks are inherently scenario-driven: they require identifying task-relevant entities, composing intermediate spatial analyses, and deriving conclusions that are actionable under application context. For example, flood-risk assessment screens buildings near water bodies [5], irrigation analysis quantifies cropland adjacency to water sources [6], and ecological stabilization identifies barren patches fully enclosed by forests [7]. In such cases, perception is only the entry point; the ultimate goal is decision-oriented geospatial reasoning. What distinguishes EO reasoning from generic multi-step perception is that it must satisfy standardized geospatial criteria specified in domain guidelines, such as buffer distances, adjacency rules, ratio thresholds, or topological predicates. These criteria cannot be reliably applied from perceptual outputs alone. Their interpretation depends on why the scene is observed, which entities are relevant, and how intermediate spatial products should be generated and reused. EO reasoning is therefore not merely a sequence of perception outputs and tool calls. Instead, it requires an explicit task structure that connects scenario intent, dependency-aware sub-goals, and standards-grounded spatial constraints, as illustrated in Figure 1.
Recent studies have adapted LLMs into RS agents, including RS-ChatGPT [8], RS-Agent [9], GeoLLM-Engine [10], Change-Agent [11], and Tree-GPT [12]. These systems show promise on perception-oriented tasks such as classification, localization, counting, and RS VQA. Recent benchmarks have begun to extend evaluation with execution-grounded tool use beyond perception outputs. ThinkGeo [13] studies execution-grounded multi-step tool-use trajectories for RS tasks, while Earth-Agent [14] emphasizes broad EO resource integration and large-scale tool orchestration. However, these advances still mainly improve task execution and tool coordination, rather than explicitly modeling three ingredients central to cognitive geospatial reasoning: (i) scenario-level intent, (ii) dependency-aware sub-goals, and (iii) standards-grounded spatial constraints. As a result, tool-executed reasoning trajectories may appear plausible, yet remain non-reproducible, non-verifiable, and difficult to compare across methods. Taken together, these limitations indicate that a benchmark for cognitive geospatial reasoning should satisfy three requirements. First, it should provide hierarchy-structured task specifications that connect scenario intent to dependency-aware sub-goals, rather than reducing reasoning to flat tool-call sequences. Second, it should support standards-grounded and verifiable spatial reasoning, where predicates, thresholds, and topology constraints can be checked through executable tools. Third, it should ensure cross-modality RS coverage for robust evaluation across heterogeneous EO data, including optical, SAR, and infrared imagery. Existing benchmarks satisfy these properties only partially and often in isolation. As summarized in Table ¿tbl:tab:benchmark95comparison?, prior benchmarks typically support verifiable tool execution and step-level evaluation, but rarely formulate reasoning around scenario-level intent, dependency-aware sub-goals, or standards-based spatial criteria. Moreover, cross-modal reasoning remains only partially evaluated.
To address this gap, we introduce TerraLogic, a benchmark for cognitive-level geospatial reasoning across multiple RS modalities. TerraLogic comprises 545 scenario-driven tasks spanning optical, SAR, and infrared imagery. Each task is specified in a structured form that couples standards-anchored spatial triggers (e.g., quantitative thresholds or topological predicates) with expert interpretation and verifiable tool chains, while organizing reasoning under scenario intents and dependency-aware sub-goals. Building on TerraLogic, we further propose HieraPlan, an LLM-driven agent that hierarchically organizes analytic tools and executes constraint-aware, fault-tolerant reasoning pipelines with recovery from intermediate failures. Our main contributions are summarized as follows:
We present TerraLogic, the first benchmark for hierarchical geospatial reasoning, comprising 545 tasks across optical, SAR, and infrared modalities.
We propose HieraPlan, an agentic framework that enables cognitive reasoning with hierarchical planning, robust error handling, and support for multi-modal RS scenarios.
We establish new baselines by comprehensively evaluating state-of-the-art LLMs on TerraLogic, revealing both their strengths and limitations in cognitively demanding geospatial reasoning.
2.5pt
lcccccc & & & &
(lr)2-4 & Optical & SAR & IR & & &
UnivEARTH univearth? & & & & & &
GeoLLM-Engine [10] & & & & & &
ThinkGeo [13] & & & & & &
Earth-Bench [14] & & & & & &
TerraLogic (Ours) & & & & & &
2 TerraLogic Benchmark Construction↩︎
We propose TerraLogic, a benchmark designed to assess the geospatial reasoning capabilities of tool-augmented agents powered by LLMs. TerraLogic integrates diverse imagery from optical, SAR, and infrared modalities with expert-curated knowledge and tool-augmented query pipelines, yielding 545 high-quality instances.
Each instance couples real-world imagery with structured, multi-step reasoning challenges that require both low-level perception (e.g., segmentation, detection) and high-level decision-making (e.g., urban planning, disaster assessment). Unlike prior remote sensing benchmarks that focus narrowly on perception (e.g., classification or detection), TerraLogic emphasizes end-to-end reasoning, from perception through spatial analysis to actionable conclusions, thereby providing a rigorous testbed for agents. Figure 2 illustrates six representative samples from the TerraLogic benchmark.
2.1 Geospatial Reasoning Scenarios↩︎
A central challenge in RS is bridging the gap between low-level perceptual outputs and the high-level intelligence required for real-world decision-making. To address this, we adopt a structured taxonomy of geospatial reasoning tasks, systematically organized into seven primary domains: urban planning, disaster assessment, environmental monitoring, transportation analysis, activation monitoring, maritime monitoring, and industrial sites.
2.2 Source Data Construction Pipeline↩︎
We construct TerraLogic with a two-stage pipeline (see Figure 3) that combines diverse source imagery, knowledge-augmented scenario generation, and multi-pass expert adjudication to produce 545 validated instances. The source datasets used in this pipeline are summarized in Table 1.
2.2.1 Stage 1: Knowledge-augmented scenario generation.↩︎
We obtain a stratified sample of source imagery (by modality and scene type) and apply explicit hardness controls to select cases that require multi-tool reasoning and compositional spatial relations (e.g., proximity, containment, topology). Candidate queries and tool-chains are generated with ChatGPT-5 via in-context learning: prompts are modality-specific and seeded with expert-authored exemplars.
To ground generation in domain knowledge, we inject a compact knowledge corpus into prompts. The corpus covers four guidance categories: (i) urban greening and heat-mitigation frameworks [15]–[19]; (ii) international land-cover standards [20], [21]; (iii) aviation and maritime search-and-rescue doctrine for IR small-target tasks [22]; and (iv) industrial safety rules for separation and proximity of hazardous assets [23], [24]. These anchors force the ChatGPT-5 to produce quantitative thresholds (distances, areas, class compositions) consistent with established frameworks. We also enforce operator legality checks (including ground distance sampling (GSD)-aware units) to prevent invalid tool sequences.
2.2.2 Stage 2: Expert adjudication and quality gating.↩︎
We perform expert-led curation with an execution-gated release criterion. Specifically, each Stage 1 candidate is subjected to a three-round vetting procedure by eight remote-sensing experts, aiming to produce instances that are both unambiguous in intent and verifiable through tool execution. Checklist-based assessment. For every candidate, reviewers examine three aspects: (i) semantic soundness (the query states a well-posed, domain-relevant task without underspecification), (ii) workflow validity (the operator choices, ordering, and arguments comply with modality constraints and remain consistent under GSD-aware units), and (iii) geospatial correctness (no geometry/topology pitfalls, and intermediate spatial objects are meaningful for downstream operators). We iteratively revise each candidate until all criteria are satisfied; otherwise, we discard it if the intended question cannot be substantiated by executable tool evidence (invalid units, ill-defined spatial predicates, or inconsistencies between the query and the expected tool outputs).
Execution-gated verification. To avoid releasing instances that rely on free-form generation, we require that the finalized tool sequence can be run to completion under operator legality constraints (typed I/O, unit validity, and geospatial constraints). We accept an instance only when (a) the tool chain executes without violations, (b) intermediate outputs pass basic validity checks (non-empty outputs where required and valid topology when applicable), and (c) any reported quantities in the final answer are directly computed from tool outputs rather than inferred without evidence. This gating criterion ensures that every released instance is paired with an executable tool chain that supports traceable, evidence-grounded evaluation.
Rounds and traceability. The three rounds consist of an initial screening, disagreement resolution for borderline cases, and a final conformance check across the query text, tool arguments, and produced outputs. All modifications to the query, operator selection, arguments, and final answer are logged in the curation interface, preserving instance-level traceability for verification and subsequent analysis.
| Name | Annotation Type | Resolution | Modality | Geographical Coverage |
|---|---|---|---|---|
| LoveDA [25] | Masks | 0.3 m/px | Optical | Nanjing/Changzhou/Wuhan, China |
| ISPRS Potsdam [26] | Masks | 0.05 m/px | Optical | Potsdam, Germany |
| ISPRS Vaihingen [27] | Masks | 0.09 m/px | Optical | Vaihingen, Germany |
| DeepGlobe Land Cover [28] | Masks | 0.50 m/px | Optical | Cities in India/Indonesia/Thailand |
| HRSCD [29] | Change masks | 0.45 m/px | Optical | Rennes/Caen, France |
| OGSOD [30] | Bounding boxes | 3 m/px | SAR | Cities in China |
| DMIST [31] | Bounding boxes | – | IR | Chengdu, China |
4pt
2.3 Benchmark Statistics↩︎
TerraLogic contains 545 queries across optical, SAR, and IR modalities. In total, 13 tools are invoked 1649 times, with most queries requiring multi-tool composition. Query lengths range from short factual questions to complex compositional ones, demonstrating both diversity and reasoning depth. Detailed statistics are reported in Table 2. Each reasoning task in TerraLogic is instantiated as a distinct scene context type, characterized by three components: (i) a Spatial Trigger (quantitative thresholds or topological relations), (ii) an Expert Interpretation (domain-grounded semantics), and (iii) a Verifiable Tool-Chain. This hierarchical design bridges perception and cognition, offering structured priors and explicit multi-step pathways while ensuring interpretability and verifiability. By covering both canonical and high-complexity reasoning scenarios, TerraLogic provides rigorous testbeds for evaluating diverse agentic capabilities, including fine-grained spatial understanding, multi-step tool composition, quantitative analysis, and context-aware risk assessment.
| Metric | All | Optical | SAR | IR |
|---|---|---|---|---|
| Total queries | 545 | 386 | 116 | 43 |
| Total tool calls | 1649 | 1217 | 346 | 86 |
| 2/3/4-tools use | 72/387/86 | 27/273/86 | 2/114/0 | 43/0/0 |
| Maximum question length | N/A | 406 | 207 | 288 |
| Minimum question length | N/A | 109 | 111 | 135 |
| Average question length | 158.24 | 160.13 | 143.49 | 181.05 |
4pt
3 The HieraPlan Agent↩︎
To support evaluation on TerraLogic, we provide HieraPlan, a hierarchy-aware and fault-tolerant agent for geospatial reasoning in remote sensing. Given a task, HieraPlan retrieves a task-relevant subset of tools from a hierarchical library spanning Perception, Spatial-Relations, Spatial-Statistics, SAR, and IR, then generates a modality-aware plan, executes concrete operators, and synthesizes the final answer from validated tool outputs. HieraPlan supports hierarchical abstraction: tools are organized by function and modality, with typed inputs/outputs and basic legality constraints, reducing invalid compositions over heterogeneous RS data. Its executable operators are grounded in remote-sensing expert models, including RemoteSAM [32], SARATR-X [33], [34], DMIST/LASNet [31], and Change3D [35]. It further enables fault-tolerant execution: when a step fails, the agent first attempts within-toolkit substitution and, if needed, replans only the remaining suffix while preserving the validated prefix. Finally, it enforces verifiable grounding: quantitative outputs are computed by tools, while the LLM is restricted to natural-language synthesis from validated intermediate results. Overall, HieraPlan serves as a suitable reference baseline for TerraLogic, whose tasks require scenario-aware decomposition, cross-modal tool coordination, and reliable execution under long reasoning chains.
4 Experiments and Results↩︎
To evaluate the reasoning and tool-use capabilities of HieraPlan under real-world RS scenarios, we conduct comprehensive experiments on the TerraLogic benchmark. Since HieraPlan is controlled by an LLM, we evaluate a diverse set of foundation models, including API-based systems—GPT-3.5 [36], GPT-4o [37], and Gemini-2.5-Flash [38]—as well as a broad range of open-source LLMs. The open-source models include large-capacity systems, such as DeepSeek-V3 [39], Qwen2.5-32B-Instruct [40], and LLaMA-3-70B-Instruct [41], together with medium-sized models, including Qwen2.5-7B [42], InternLM3-8B [43], LLaMA-3-8B [41], Mistral-7B [44], Yi-1.5-6B [45], and Phi-3-Mini [46]. All experiments are conducted on an NVIDIA A5000 GPU within the OpenCompass [47] evaluation platform.
4.1 Overall Performance↩︎
We evaluate HieraPlan on the TerraLogic benchmark under a ReAct-style [48] protocol with both step-by-step and end-to-end settings. The step-by-step metrics—InstAcc, ToolAcc, ArgAcc, and SummAcc—measure instruction following, tool selection, argument correctness, and summary generation, respectively. The end-to-end scores—P, O, and L—assess tool-augmented predictions on the Perception, Spatial-Statistics, and Spatial-Relations toolkits. Within this setup, HieraPlan complements rather than alters the ReAct protocol, providing hierarchical orchestration while preserving the same evaluation criteria. For final answer accuracy (AnsAcc), we avoid deterministic string matching (e.g., GTA [49]), which is brittle to minor lexical variations and may penalize semantically correct outputs. This is especially important in TerraLogic, where final answers are synthesized from query semantics, tool inputs/outputs, and scenario knowledge, rather than matched to a single canonical string. Following prior work on LLM-as-a-judge [50]–[52], we compute AnsAcc using GPT-4o-mini with a rubric-based prompt to assess whether the predicted answer and its key arguments are semantically consistent with the ground truth. Table 3 reports the overall results on TerraLogic, whose multimodal, scenario-driven tasks require agents to coordinate multiple toolkits spanning perception, spatial analysis, SAR, and IR, thereby enabling systematic evaluation of cognitive-level geospatial reasoning. As shown in Table 3, GPT-4o achieves the strongest overall performance across both evaluation settings, leading on InstAcc, ArgAcc, SummAcc, as well as P, O, and Ans. Among open-source large models, Llama-3-70B-Instruct performs best overall, with competitive ToolAcc and strong end-to-end results. Among medium-scale open-source models, Qwen2.5-7B-Instruct is notable for achieving the highest L score (Spatial Relations) across all evaluated models, despite weaker InstAcc and SummAcc. Across model families, ArgAcc remains the main bottleneck, while lower ToolAcc is consistently associated with reduced end-to-end performance, highlighting the difficulty of reliable tool selection and faithful argument propagation in long-horizon geospatial workflows.
| Model | Step-by-Step Metrics (%) | End-to-End Metrics (%) | ||||||
| Inst. | Tool. | Arg. | Summ. | P | O | L | Ans. | |
| API-based | ||||||||
| GPT-4o | 79.40 | 72.01 | 23.57 | 82.83 | 72.73 | 88.38 | 77.95 | 25.70 |
| Gemini-2.5-Flash | 59.09 | 43.31 | 15.75 | 47.22 | 41.47 | 40.89 | 41.51 | 12.04 |
| GPT-3.5 | 59.05 | 57.12 | 21.88 | 59.08 | 61.22 | 61.04 | 53.30 | 24.40 |
| Open-source (Large) | ||||||||
| DeepSeek-V3 | 65.84 | 66.33 | 23.32 | 60.92 | 65.44 | 66.04 | 57.93 | 21.28 |
| Qwen2.5-32B-Instruct | 62.83 | 58.00 | 17.41 | 58.90 | 44.28 | 67.40 | 61.47 | 19.08 |
| Llama-3-70B-Instruct | 67.89 | 67.47 | 22.66 | 69.91 | 62.72 | 71.62 | 62.54 | 22.75 |
| Open-source (Medium/Small) | ||||||||
| Qwen2.5-7B-Instruct | 77.19 | 74.71 | 23.11 | 65.32 | 68.56 | 73.03 | 82.42 | 22.57 |
| InternLM3-8B-Instruct | 61.26 | 57.72 | 22.59 | 50.28 | 61.44 | 53.70 | 60.70 | 18.90 |
| LLaMA3-1-8B | 69.46 | 65.99 | 22.27 | 55.60 | 68.42 | 60.34 | 70.82 | 16.70 |
| Phi-3-Mini-4K-Instruct | 55.45 | 41.14 | 17.17 | 10.09 | 43.63 | 7.52 | 51.74 | 3.67 |
| Mistral-7B-Instruct-v0.2 | 60.03 | 47.05 | 16.73 | 46.06 | 49.75 | 44.05 | 45.23 | 13.21 |
| Yi-1.5-6B-Chat | 63.38 | 42.08 | 16.00 | 37.98 | 47.44 | 38.73 | 36.02 | 10.83 |
4.2 Task Error Analysis.↩︎
To explain the performance gaps above, we analyze failures using an error taxonomy covering format, reasoning, perception, and tool use. As shown in Table 4, format errors are the dominant failure mode across model families. They are particularly severe for open-source models, including DeepSeek-V3 (66.40%), Llama-3-70B (55.20%), and Phi-3-Mini (45.60%), but remain substantial even for GPT-4o (41.65%). This suggests that generating valid structured tool calls, especially well-formed JSON arguments, remains a major bottleneck. Tool-use errors further distinguish model classes. GPT-4o is the most reliable (3.60%), whereas Gemini-2.5-Flash (37.26%) and GPT-3.5 (39.50%) more often select inappropriate operators or produce invalid parameters. Some open-source models show lower tool-use error rates, but this likely reflects more conservative behavior rather than stronger execution. Another major source of failure is perception grounding. For example, GPT-4o shows a perception error rate of 41.26%, indicating that interpreting land-cover masks, change maps, and spatial layouts remains challenging. By contrast, reasoning errors are relatively moderate across models (13–35%) and do not appear to be the main bottleneck. Overall, these results suggest that current geospatial agents are constrained more by structured output generation and multimodal grounding than by abstract reasoning alone.
| Model | Format | Reasoning | Perception | Tool-use |
|---|---|---|---|---|
| GPT-4o | 41.65 | 13.49 | 41.26 | 3.60 |
| Gemini-2.5-Flash | 29.47 | 21.18 | 12.09 | 37.26 |
| GPT-3.5 | 31.20 | 18.60 | 10.70 | 39.50 |
| DeepSeek-V3 | 66.40 | 23.00 | 8.70 | 1.90 |
| Qwen2.5-32B | 31.27 | 19.78 | 13.99 | 34.96 |
| Llama-3-70B | 55.20 | 25.70 | 18.20 | 0.90 |
| Qwen2.5-7B | 42.65 | 23.48 | 29.37 | 4.50 |
| InternLM3-8B | 32.70 | 23.90 | 11.40 | 32.00 |
| LLaMA3.1-8B | 38.36 | 24.58 | 20.28 | 16.78 |
| Phi-3-Mini-4K | 45.60 | 34.70 | 1.80 | 17.90 |
| Mistral-7B-v0.2 | 45.40 | 21.20 | 5.40 | 28.00 |
| Yi-1.5-6B | 37.90 | 28.60 | 8.60 | 24.90 |
4.3 Comparison with Remote Sensing Multimodal LLMs↩︎
To contextualize TerraLogic’s difficulty, we compare HieraPlan with existing remote-sensing multimodal LLMs (RS-MLLMs) on the optical subset, where these models are expected to perform best. Table 5 reports the results. Existing RS-MLLMs perform poorly on TerraLogic’s optical geospatial reasoning tasks: EarthDial, GeoChat, and GeoPix achieve only 9.38%, 6.70%, and 6.17% accuracy, respectively, despite being designed for remote-sensing understanding. In contrast, HieraPlan (GPT-4o) reaches 29.29% accuracy, substantially outperforming these specialized RS-MLLMs. Nevertheless, the overall performance remains low, indicating that TerraLogic’s optical tasks are still highly challenging and require fine-grained spatial reasoning beyond standard VQA-style interpretation.
| Model | Optical Accuracy (%) |
|---|---|
| RS-MLLMs | |
| EarthDial[53] | 9.38 |
| GeoChat[54] | 6.70 |
| GeoPix[55] | 6.17 |
| LLM-based Agent (Ours) | |
| HieraPlan (GPT-4o) | 29.29 |
6pt
4.4 Fine-grained capability subsets.↩︎
To pinpoint where failures arise, Table 6 stratifies TerraLogic into four subsets with progressively increasing capability composition and planning horizon. Semantic + Geometry (S+G) includes tasks requiring semantic perception followed by geometric reasoning (e.g., segmentation with buffering or overlap), without quantitative statistics. Semantic + Quantitative (S+Q) combines semantic perception with numerical measurement (e.g., detection with distance estimation or counting), without geometric predicates. Semantic + Geometry + Quantitative (S+G+Q) represents full-pipeline workflows integrating all three capabilities. Long-Horizon (H) includes instances with \(K\ge4\) sequential tool operations, and we further report \(H\cap\)(S+G+Q) to isolate the most challenging long-horizon full-pipeline cases. Two trends emerge. First, capability composition is substantially harder than any individual capability: for most models, performance drops from S+G or S+Q to S+G+Q, indicating that errors accumulate when arguments must be propagated across heterogeneous toolkits. Second, planning horizon remains the dominant challenge: on \(H\cap\)(S+G+Q), even strong systems retain moderate step-level accuracies but still degrade sharply in final answer accuracy, suggesting that small local tool or argument errors can derail end-to-end correctness in long reasoning chains.
| Model | Task | Inst | Tool | Arg | Summ | Ans |
|---|---|---|---|---|---|---|
| API-based | ||||||
| GPT-4o | S+G | 77.78 | 92.11 | 32.89 | 75.00 | 45.29 |
| S+Q | 78.33 | 89.20 | 22.80 | 94.00 | 36.00 | |
| S+G+Q | 85.20 | 84.40 | 27.37 | 91.08 | 35.43 | |
| H \(\cap\) (S+G+Q) | 60.70 | 46.77 | 18.09 | 60.47 | 9.81 | |
| Gemini-2.5-Flash | S+G | 49.44 | 48.03 | 18.42 | 53.57 | 21.43 |
| S+Q | 30.67 | 25.60 | 12.00 | 30.00 | 8.00 | |
| S+G+Q | 34.51 | 23.40 | 9.00 | 24.93 | 9.97 | |
| H \(\cap\) (S+G+Q) | 50.81 | 34.24 | 12.40 | 41.86 | 0.00 | |
| GPT-3.5 | S+G | 67.20 | 73.00 | 27.00 | 64.30 | 42.90 |
| S+Q | 83.30 | 94.40 | 40.80 | 98.00 | 48.00 | |
| S+G+Q | 53.90 | 53.50 | 20.80 | 51.20 | 24.70 | |
| H \(\cap\) (S+G+Q) | 67.10 | 54.40 | 18.30 | 69.80 | 10.50 | |
| Open-source (Large) | ||||||
| DeepSeek-V3 | S+G | 80.00 | 86.80 | 28.30 | 92.90 | 32.10 |
| S+Q | 83.30 | 96.80 | 30.00 | 98.00 | 40.00 | |
| S+G+Q | 61.90 | 65.70 | 24.00 | 53.80 | 21.30 | |
| H \(\cap\) (S+G+Q) | 70.70 | 54.80 | 17.70 | 60.50 | 5.80 | |
| Qwen2.5-32B-Instruct | S+G | 59.40 | 65.10 | 23.70 | 60.70 | 10.70 |
| S+Q | 83.30 | 94.40 | 41.20 | 98.00 | 32.00 | |
| S+G+Q | 60.10 | 57.50 | 15.40 | 52.00 | 21.30 | |
| H \(\cap\) (S+G+Q) | 66.00 | 46.50 | 15.20 | 66.30 | 5.80 | |
| Llama-3-70B-Instruct | S+G | 80.00 | 80.26 | 24.34 | 89.29 | 25.00 |
| S+Q | 83.33 | 92.80 | 38.80 | 94.00 | 34.00 | |
| S+G+Q | 64.44 | 70.12 | 23.17 | 62.99 | 23.10 | |
| H \(\cap\) (S+G+Q) | 72.21 | 47.67 | 15.37 | 80.23 | 11.63 | |
| Open-source (Medium/Small) | ||||||
| Qwen2.5-7B-Instruct | S+G | 80.00 | 94.70 | 32.20 | 85.70 | 42.90 |
| S+Q | 83.30 | 96.00 | 32.40 | 100.00 | 34.00 | |
| S+G+Q | 78.70 | 77.70 | 23.50 | 57.50 | 22.00 | |
| H \(\cap\) (S+G+Q) | 69.00 | 53.60 | 17.10 | 73.30 | 12.80 | |
| InternLM3-8B-Instruct | S+G | 74.40 | 82.90 | 28.90 | 78.60 | 25.00 |
| S+Q | 83.30 | 94.40 | 41.20 | 100.00 | 32.00 | |
| S+G+Q | 55.80 | 55.20 | 22.20 | 38.60 | 19.70 | |
| H \(\cap\) (S+G+Q) | 70.20 | 49.60 | 16.80 | 64.00 | 5.80 | |
| LLaMA3-1-8B | S+G | 80.00 | 89.50 | 28.30 | 85.70 | 21.40 |
| S+Q | 51.70 | 51.60 | 28.40 | 40.00 | 6.00 | |
| S+G+Q | 71.20 | 71.00 | 23.20 | 50.70 | 19.20 | |
| H \(\cap\) (S+G+Q) | 67.30 | 48.70 | 16.00 | 76.70 | 10.50 | |
| Phi-3-Mini-4K-Instruct | S+G | 51.11 | 42.11 | 14.47 | 32.14 | 7.14 |
| S+Q | 77.67 | 82.00 | 32.40 | 72.00 | 16.00 | |
| S+G+Q | 57.35 | 40.98 | 18.34 | 2.62 | 2.62 | |
| H \(\cap\) (S+G+Q) | 41.86 | 28.29 | 8.79 | 0.00 | 0.00 | |
| Mistral-7B-Instruct-v0.2 | S+G | 73.90 | 74.30 | 24.30 | 78.60 | 25.00 |
| S+Q | 77.70 | 86.00 | 36.40 | 92.00 | 16.00 | |
| S+G+Q | 56.00 | 43.60 | 15.10 | 37.30 | 13.60 | |
| H \(\cap\) (S+G+Q) | 65.30 | 40.80 | 14.30 | 47.70 | 5.80 | |
| Yi-1.5-6B-Chat | S+G | 81.67 | 63.82 | 23.68 | 85.71 | 14.29 |
| S+Q | 54.33 | 56.40 | 21.20 | 38.00 | 14.00 | |
| S+G+Q | 60.37 | 41.02 | 15.49 | 30.97 | 11.02 | |
| H \(\cap\) (S+G+Q) | 73.37 | 36.82 | 14.60 | 53.49 | 6.98 | |
2.2pt
5 Conclusion↩︎
In this work, we introduced TerraLogic, the first benchmark for cognitive-level geospatial reasoning across optical, SAR, and IR modalities. TerraLogic comprises 545 hierarchy-aware, scenario-driven tasks that extend evaluation beyond perception to structured, multi-step analysis. To support benchmark evaluation, we further provide HieraPlan, a hierarchy-aware, fault-tolerant baseline agent for multimodal tool-augmented reasoning. Experiments on TerraLogic show that even frontier models remain limited, with clear weaknesses in argument prediction, summary generation, and multimodal tool integration, highlighting substantial headroom for future research on geospatial reasoning, planning, and execution.