1 Introduction↩︎

Remote sensing image scene understanding (RSISU) aims to interpret complex earth observation scenes beyond image-level recognition. It covers scene classification (SC), image captioning (IC), visual question answering (VQA), and visual grounding (VG). It also involves object localization, attribute analysis, spatial reasoning, and change understanding [1]–[3]. Remote sensing (RS) images differ substantially from natural images. They are characterized by overhead viewpoints, large variations in object scale, dense object distributions, and diverse sensor modalities. Their interpretation also depends strongly on spatial resolution, viewing geometry, illumination conditions, and geographic context. Models must recognize individual objects and land-cover categories while understanding their spatial relationships, scene-level semantics, and geographic context. Although recent foundation models (FMs) and multimodal large language models (MLLMs) have significantly advanced general visual understanding, these RS-specific challenges remain difficult to address.

Foundation models are large-scale pre-trained models that can be adapted to downstream tasks through fine-tuning, few-shot learning, or zero-shot transfer. Representative examples include language FMs such as GPT-3 [4], vision FMs such as MAE [5], and vision-language FMs such as CLIP [6] and BLIP [7]. Early RS FMs mainly focused on visible optical imagery. Models such as SatMAE [8], ScaleMAE [9], RingMo [10], and GFM [11] learned transferable representations from large-scale RS image collections. More recent RS FMs have expanded beyond optical imagery to hyperspectral, synthetic aperture radar (SAR), and multimodal observations. SpectralGPT [12] and HyperSIGMA [13] focus on hyperspectral representation learning, whereas SARJEPA [14] and SARMAE [15] target SAR imagery. SkySense [16] and TerraFM [17] further explore unified representations across multiple RS modalities. These models provide important visual and cross-modal backbones for the development of RS-oriented MLLMs.

MLLMs extend conventional large language models (LLMs) with multimodal perception and cross-modal reasoning capabilities. They support instruction-following tasks such as VQA, image captioning, visual grounding, image-based dialogue, and multimodal reasoning [18]. A typical MLLM consists of a visual encoder, a modality alignment module, and an LLM. The visual encoder extracts image features, while the alignment module maps these features into representations that can be processed by the LLM. This design enables more flexible interaction than conventional task-specific vision models. Proprietary systems such as GPT-4V and Gemini have demonstrated strong multimodal reasoning capabilities. Open-source models such as mPLUG-Owl have further demonstrated the potential of adapting pre-trained knowledge to multimodal tasks at reduced training cost [19]. These advances create new opportunities for RSISU, which requires the joint modeling of visual perception, geographic semantics, and natural-language interaction.

Early studies explored the direct application of general-purpose MLLMs and tool-augmented systems to RS and geospatial tasks. [20] evaluated Visual ChatGPT on scene classification, edge detection, line detection, and segmentation. The model showed potential in relatively simple scenes but performed poorly in complex urban environments. Similarly, [21] examined several language, vision, and multimodal foundation models for geospatial artificial intelligence. These models performed well on text-based geospatial tasks under zero-shot and few-shot settings. However, they remained limited when tasks required the integration of multiple data modalities [22]. [23] further evaluated GPT-4V on Earth observation tasks. GPT-4V achieved promising results in aerial landmark recognition and image captioning but struggled with object localization, counting, spatial reasoning, and change detection. These early findings revealed several limitations in applying general-purpose MLLMs to RSISU and motivated the development of RS-specific models. However, general-purpose MLLMs have since advanced rapidly in visual perception and multimodal reasoning. This progress raises a central question: Do RS-specific MLLMs consistently outperform general-purpose models across diverse RSISU tasks?

In this survey, MLLMs primarily developed for general visual understanding are referred to as CV-MLLMs. Models specifically designed or adapted for RS tasks are referred to as RS-MLLMs. Early RS-MLLMs mainly focused on image captioning and VQA. More recent models have expanded toward multitask dialogue, visual grounding, spatial reasoning, and large-scale scene understanding [24]. Representative models include RSGPT [25], GeoChat [26], SkyEyeGPT [27], EarthGPT [28], LHRS-Bot [29], H²RSVLM [30], and SkySenseGPT [31]. These studies have contributed to RS instruction tuning, multimodal dialogue, grounded understanding, dataset construction, and complex scene interpretation. In parallel, increasing attention has been given to RS-oriented vision encoders and vision-language foundation models. These components are important for handling the sensor diversity, scale variation, and geographic semantics of RS imagery [32], [33].

Recent RS-MLLMs have further advanced toward finer-grained, sensor-aware, and reasoning-oriented understanding. Falcon [34] constructs a large-scale multitask instruction-tuning corpus to support image-, region-, and pixel-level interpretation. GeoLLaVA-8K [35] and ZoomEarth [36] focus on ultra-high-resolution RS imagery. They employ high-resolution encoding, adaptive cropping, or active perception to process large scenes. EarthGPT-X [37] extends RS-MLLMs to multi-source spatial understanding by integrating optical, SAR, and infrared imagery with visual prompting. Geo-R1 [38] introduces reinforcement fine-tuning for few-shot geospatial referring expression understanding. RemoteShield [39] investigates robustness against realistic visual and textual perturbations. Collectively, these advances show that RS-MLLM research is moving beyond conventional captioning and VQA. Emerging priorities include high-resolution perception, spatial grounding, multi-source interpretation, reasoning-oriented post-training, and reliable deployment.

Despite this progress, existing surveys have not yet provided a systematic and diagnostic understanding of RS-MLLMs for RSISU. Most reviews focus on RS foundation models, vision-language datasets, or general MLLM architectures [40]. The effects of RS-specific instruction tuning, RS-oriented visual encoders, and multimodal alignment remain insufficiently examined. Existing models are also evaluated using different datasets, prompts, metrics, and inference protocols. These differences make direct comparisons difficult. More importantly, it remains unclear whether RS-specific instruction tuning provides consistent advantages over recent general-purpose MLLMs. This issue is increasingly important because CV-MLLMs have rapidly improved in visual perception and cross-modal reasoning. In contrast, many RS-MLLMs are trained with relatively limited instruction data and evaluated under heterogeneous protocols.

As summarized in Table 1, this survey differs from previous reviews by combining a systematic taxonomy with a diagnostic evaluation. We examine the technical development of RS-MLLMs and analyze the roles of model architecture, visual encoders, multimodal alignment, instruction tuning, and training data. We further compare RS-MLLMs with recent open-source and proprietary CV-MLLMs under representative RSISU settings. Our analysis investigates whether domain-specific adaptation leads to consistent performance gains. The results reveal that the relative strengths of RS-MLLMs and CV-MLLMs vary considerably across tasks. RS-MLLMs remain competitive in several domain-specific settings. More notably, general-purpose CV-MLLMs can match or even outperform specialized RS-MLLMs on several RSISU tasks without remote sensing-specific fine-tuning. This diagnostic perspective provides an empirical basis for future model development, benchmark construction, and evaluation protocol design. We focus primarily on end-to-end and instruction-tuned RS-MLLMs, while discussing RS agents as an important direction toward executable and verifiable geospatial intelligence.

We searched Google Scholar, Web of Science, IEEE Xplore, Scopus, Elsevier, Springer, and arXiv for relevant studies published through July 2026. The search queries included “remote sensing multimodal large language model,” “remote sensing vision–language model,” “Earth observation multimodal model,” “remote sensing agent,” and combinations of these terms. We included studies that developed, adapted, or systematically evaluated instruction-following multimodal models for remote sensing image understanding. Studies limited to unimodal representation learning, conventional image classification, or language-only geospatial tasks were excluded unless they provided essential architectural, methodological, or historical context for the development of RS-MLLMs.

Our main findings and contributions are summarized as follows:

• We provide a systematic review of RS-MLLMs for RSISU. The review covers model architectures, key components, instruction-tuning data, training strategies, and evaluation settings within the broader development of visual and vision-language foundation models.

• We systematically compare state-of-the-art RS-MLLMs with open-source and proprietary CV-MLLMs across representative RSISU datasets and tasks. This comparison provides a diagnostic assessment beyond the results reported by individual studies.

• We show that the relative performance of RS-MLLMs and CV-MLLMs is strongly task-dependent. General-purpose CV-MLLMs can match or outperform specialized RS-MLLMs on several RSISU tasks without RS-specific fine-tuning. We also identify limitations related to instruction diversity, evaluation protocols, and potential benchmark contamination.

• We discuss future directions for reliable evaluation, spatially grounded reasoning, training-free adaptation, multimodal and multitemporal understanding, high-resolution image processing, efficient deployment, and RS agents.

The remainder of this survey is organized as follows. Section 2 presents the taxonomy of RS-MLLMs and reviews their instruction-tuning datasets, model architectures, training strategies, and key characteristics. Section 3 compares RS-MLLMs and CV-MLLMs across representative RSISU tasks. Section 4 discusses the main challenges and future research directions. Finally, Section 5 concludes the survey.

3pt

2 Taxonomy of RS-MLLMs↩︎

This section will outline these RSISU-specific challenges and subsequently review the current landscape of RS-FMs, RS-MLLMs and RS-Agents. Following this discussion, a detailed analysis of the model architectures and datasets supporting existing RS-MLLM research will be presented.

2.1 RSISU Requirements↩︎

Unique requirements of RSISU. Successfully adapting MLLMs for RS applications necessitates addressing the following three primary aspects. 1) Data characteristics: MLLMs must be equipped to process multi-resolution, multimodal, multi-platform, and multi-temporal imagery from various viewpoints. This capability involves handling diverse atmospheric conditions and integrating temporal data for change detection. Additionally, these models must incorporate geospatial information to enhance contextual understanding. 2) Task-specific requirements: RSISU applications require fine-grained object recognition, particularly for small and diverse objects and advanced spatial reasoning to interpret complex geographical layouts and relationships. RS-MLLMs must be optimized for specialized tasks, such as land use classification and crop monitoring while ensuring explainable outputs to support decision-making processes. 3) Domain knowledge integration: RS-MLLMs need to possess a comprehensive understanding of RS-specific features, terrain types, and man-made structures. Furthermore, these models must comprehend the broader geographical and environmental context of observed features.

2.2 Remote Sensing FMs↩︎

As summarized in Table 2, the development of RS FMs exhibits a clear progression from single-modality visual pretraining to multi-modal and vision-language representation learning. Early RS FMs mainly focus on visible optical imagery, where models such as Geograph [46], RSP [47], SatMAE [8], ScaleMAE [9], RingMo [10], Billion [49], and GFM [11] learn transferable visual representations from large-scale RS images. Subsequently, RS FMs have been extended to more diverse Earth observation modalities. SpectralGPT [12], HyperSIGMA [13], and HyperFree [50] target spectral imagery, while SARJEPA [14], SARATR-X [51], and SARMAE [15] focus on SAR representation learning. More recent models, such as SkySense [16], TerraFM [17], and SkySense V2 [52], further pursue unified representations across multiple RS modalities, reflecting the heterogeneous and multi-source nature of Earth observation.

Another important trend is the transition from purely visual representation learning to vision-language pretraining. Methods such as SkyCLIP [32], RemoteCLIP [53], GRAFT [33], GeoRSCLIP [54], SARVLM [57], SARCLIP [58], and HyperCap [59] align RS imagery with natural-language descriptions, enabling cross-modal retrieval, zero-shot recognition, and downstream RS vision-language tasks. These visual and vision-language FMs provide important encoder candidates for RS-MLLMs. However, most existing RS FMs are still designed for representation learning or cross-modal alignment, rather than interactive reasoning, multi-image understanding, instruction following, and complex RSISU tasks. Therefore, how to effectively integrate RS-specific foundation encoders with powerful LLM backbones remains a key issue for building more capable RS-MLLMs.

2.3 Remote Sensing MLLMs↩︎

RS-MLLMs [62]–[64] continue to evolve rapidly, as illustrated in Figure 2. RSGPT represents a pioneering integration of LLMs with RS tasks, focusing on IC and VQA [25]. By adopting the InstructBLIP model, RSGPT demonstrates notable performance in scene description tasks and introduces a dense captioning dataset known as RSICap. However, despite its strengths, RSGPT requires task-specific tuning, which limits its generalizability. It also faces challenges in handling diverse tasks, including region-level reasoning, and specific tasks such as classification, detection, and visual grounding.

GeoChat extends conversational capabilities within the remote sensing domain by enabling region-specific dialogues and visual grounding [26]. Built upon the LLaVA1.5 architecture, GeoChat effectively handles both image-level and region-specific queries, grounding objects in images using spatial coordinates. However, despite advancements in spatial reasoning, GeoChat encounters challenges in tasks requiring fine-grained recognition, such as detailed object recognition and counting.

SkyEyeGPT unifies visual-language tasks at both image and region levels, utilizing an architecture based on MiniGPT-v2 [27]. It employs a two-stage tuning method to enhance instruction-following and multi-turn dialogue capabilities across varying levels of granularity. Notably, SkyEyeGPT is the first model to support RS video captioning tasks, marking a significant step in expanding RS-MLLM functionalities.

EarthGPT focuses on the unification of multi-sensor RS tasks and the enhancement of RS expert knowledge [28]. By leveraging the large-scale dataset MMRS-1M, which comprises diverse sensor images such as optical, synthetic aperture radar (SAR), and infrared, EarthGPT emphasizes cross-modal comprehension. This inclusion of SAR and infrared images broadens the model’s input capabilities and supports more versatile RS applications.

LHRS-Bot employs an innovative multi-level vision-language alignment strategy complemented by a curriculum learning approach. It introduces the LHRS-Align and LHRS-Instruct datasets, leveraging geographical information to advance understanding and reasoning in RSISU tasks. This methodology enhances the model’s ability to process complex spatial information effectively.

RS-CapRet specializes in IC and text–image retrieval for RS tasks [65]. It utilizes a straightforward training procedure that aligns a visual encoder with a language model and can manage interleaved image-text dialogues. While innovative, RS-CapRet has a narrower focus compared to other models that address a broader range of tasks. RS-LLaVA emphasizes multitasking capabilities within RS imagery, employing a low-rank adaptation approach primarily focused on IC and VQA [60].

H²RSVLM stands out by introducing trustworthiness as a central concept in RS models, enhancing the honesty and helpfulness of RS-MLLMs through targeted datasets. It tackles the prevalent hallucination problem by developing resources that strengthen the model’s self-awareness and its ability to decline responses to unanswerable questions.

SkySenseGPT advances relation reasoning and comprehension tasks through a large-scale instruction tuning dataset [31]. Utilizing graphs to improve perception and relational reasoning in RS imagery addresses the template limitations of earlier models. Its significant contribution lies in managing complex relationships critical for fine-grained scene understanding, moving from basic relation reasoning to image-level and region-level graph generation. RS-GPT4V introduces the RS-GPT4V dataset to unify multimodal tasks and hierarchical instruction descriptions, focusing on reasoning and detailed scene comprehension [61]. By employing strategies such as local and global hierarchical instruction descriptions and multi-turn question-answer pairs, RS-GPT4V captures fine-grained information and implicit knowledge across diverse RS tasks, surpassing existing datasets in coverage and capability.

Falcon presents a holistic RS vision-language foundation model with unified prompt-based outputs across image-, region-, and pixel-level tasks [34]. Supported by the large-scale Falcon SFT dataset, which contains about 78 million samples from 5.6 million RS images, Falcon is capable of handling 14 tasks, including IC, VQA, counting, captioning, detection, grounding, segmentation, and change detection. Compared with previous RS-MLLMs that mainly focus on a limited set of image- or region-level tasks, Falcon significantly expands the functional coverage of RS vision-language modeling. However, despite its comprehensive task design, Falcon is still mainly trained and evaluated within predefined task templates, and its open-ended reasoning ability in complex real-world RS scenarios requires further investigation.

Geo-R1 further explores the potential of reasoning- oriented reinforcement fine-tuning in RS-MLLMs, especially under few-shot geospatial referring expression understanding scenarios [38]. Different from conventional SFT-based paradigms, Geo-R1 adopts GRPO-based reinforcement fine-tuning to encourage explicit reasoning chains before producing spatial predictions. By constructing VRS- Bench-FS, NWPU-FS, and EarthReason-FS, Geo-R1 provides a standardized few-shot evaluation protocol for REC, OVD, and GRES tasks. Its main contribution lies in improving spatial grounding and cross-dataset generalization with limited annotated samples. However, its application scope is mainly concentrated on referring expression understanding, while broader multimodal RS tasks such as captioning, general VQA, and multi-sensor interpretation remain less explored.

ZoomEarth addresses the challenge of ultra-high resolution RS image understanding by introducing an active perception paradigm [36]. Instead of passively processing the whole image at a fixed resolution, ZoomEarth adaptively invokes a cropping–zooming tool to revisit informative regions and conduct fine-grained reasoning. Built upon the LRS-GRO benchmark, it covers global-, region-, and object-level questions and employs SFT followed by GRPO with a Region-Guided reward. This design enhances the model’s ability to localize regions of interest and reason over detailed visual contents in UHR imagery. Nevertheless, ZoomEarth relies on tool-based region exploration, and its effectiveness may depend on the accuracy of region selection and the quality of cropped visual evidence.

GeoLLaVA-8K focuses on scaling RS-MLLMs to ultra-high-resolution inputs up to 8K\(\times\)​8K [35]. To address the lack of UHR vision-language data, it introduces SuperRS-VQA and HighRS-VQA, two high-resolution RS VQA datasets covering diverse real-world dialogue tasks. Meanwhile, to mitigate the token explosion problem caused by large image sizes, GeoLLaVA-8K proposes Background Token Pruning and Anchored Token Selection, preserving semantically important tokens while reducing computational cost. This enables effective UHR RS understanding and achieves strong performance on XLRS-Bench. However, its current focus is mainly on optical satellite imagery and VQA-style tasks, leaving broader sensor modalities and more diverse RS reasoning tasks for future exploration.

2.4 Remote Sensing Agents↩︎

With the rapid development of agent technologies [66]–[68] in the general computer vision community, remote sensing agents have also begun to attract increasing research attention. Remote sensing agents represent a recent extension of RS-MLLMs from passive vision-language interaction toward executable geospatial intelligence. Unlike conventional RS-MLLMs that mainly answer questions or generate descriptions from input images, RS agents aim to understand user instructions, plan multi-step workflows, invoke external tools, and integrate intermediate results for final decision making. This paradigm is particularly important for RSISU because many real-world remote sensing tasks require not only visual perception, but also geospatial data access, object detection, segmentation, change detection, spatial computation, retrieval, and domain-specific analysis.

Tool-augmented RS agents. The first line of development focuses on tool-augmented RS agents. Instead of relying only on the internal perception and reasoning ability of a single MLLM, these systems use an LLM or MLLM as the central controller to decompose user instructions and call specialized tools. RS-Agent introduces a central controller, dynamic toolkit, solution space, and knowledge space to automate remote sensing tasks, while task-aware retrieval and DualRAG improve tool selection and domain knowledge grounding [69]. GeoLLM-Engine further builds a realistic environment for geospatial copilots by incorporating geospatial APIs, dynamic maps, user-interface states, and external multimodal knowledge sources [70]. ThinkGeo evaluates tool-augmented agents on structured remote sensing tasks involving optical and SAR imagery, emphasizing multi-step planning and tool execution [71]. These studies show that RS agents can connect MLLMs with detectors, segmentation models, change detection algorithms, GIS functions, and knowledge bases, thereby extending RS-MLLMs from image-level understanding to remote sensing workflows.

Benchmark and process-level evaluation. A second line focuses on benchmark construction and process-level evaluation for RS agents. Existing RS-MLLM benchmarks usually evaluate final answers on static image-text pairs, which are insufficient for agentic systems that interact with tools and dynamic platforms. GeoLLM-QA argues that RS agent evaluation should consider long sequences of verbal, visual, and click-based actions on real remote sensing platforms, where user intent may depend on map locations, selected regions, and UI states [72]. ThinkGeo similarly evaluates not only final-answer correctness, but also step-wise execution quality, tool accuracy, and planning consistency, which are critical for diagnosing failures in long-horizon RS workflows [71]. Earth-Agent and Earth-Bench further strengthen this direction by evaluating both reasoning trajectories and final results for EO tasks requiring domain-specific tools and multi-step quantitative analysis [73]. These works suggest that future RS agent benchmarks should measure whether the agent selects the right tools, follows reasonable intermediate steps, and produces verifiable conclusions.

Multi-agent geospatial copilots. A third direction is multi-agent geospatial copilots. Complex RS workflows often involve heterogeneous data sources, multiple analytical stages, and different types of expertise, making it difficult for a single monolithic agent to handle all subtasks robustly. GeoLLM-Squad addresses this issue by separating agentic orchestration from task-specific geospatial problem solving and delegating subtasks to specialized agents [74]. Such a design naturally supports roles such as planner, retriever, vision worker, GIS worker, and verifier, and can improve scalability for applications such as urban monitoring, forestry protection, climate analysis, and agricultural studies. This multi-agent paradigm indicates that RS agents may evolve from single-controller systems toward collaborative geospatial copilot teams.

Domain-specific and professional RS agents. A fourth direction moves RS agents toward domain-specific and professional scenarios. MineAgent targets remote-sensing mineral exploration by combining geological knowledge, hyperspectral information, and multi-image reasoning [75]. REMSA focuses on automatic remote sensing foundation model selection, using an LLM agent to interpret user requirements and rank suitable RSFMs from a structured model database [76]. ShapefileGPT extends agentic geospatial analysis to vector GIS data by using a multi-agent framework for automated Shapefile processing and spatial queries [77]. These studies suggest that future RS agents will increasingly move from general image understanding toward professional workflows such as mineral exploration, disaster assessment, foundation model recommendation, and vector GIS analysis.

Active tool exploration and unified multimodal reasoning. More recent studies further address scalability in tool organization and multimodal reasoning. RS-Claw tackles the context burden of large tool libraries by organizing tools into hierarchical skill trees and enabling progressive active tool exploration, thereby reducing token overhead while preserving access to critical tools [78]. In parallel, RingMo-Agent follows a model-centric route by building a unified multi-platform and multi-modal RS foundation model over optical, SAR, and infrared imagery from satellite and UAV platforms [79]. Overall, the development path of RS agents is becoming clear: from single-model RS-MLLMs, to tool-augmented agents, to realistic platform evaluation, multi-agent collaboration, domain-specific applications, and active tool exploration. Despite this progress, RS agents still face challenges in reliable tool selection, long-horizon reasoning, trajectory verification

3.5pt

Risk definition: Low indicates no identified overlap and an eligible pre-release checkpoint; Medium indicates post-release evaluation or incomplete training-data disclosure without confirmed overlap; High indicates explicit use of the benchmark, its source images, training split, few-shot subset, or task annotations.

2.5 Instruction Tuning Datasets↩︎

In this paragraph, as shown in Table 4, the instruction tuning datasets used in representative RS-MLLMs are illustrated. Existing datasets are generally constructed by reorganizing public RS benchmarks and enriching them with captions, question-answer pairs, grounding annotations, or multi-turn conversations. These datasets cover diverse tasks, including scene classification, image captioning, VQA, visual grounding, object detection, region-level captioning, reasoning, segmentation, and change detection. Recent works further extend instruction tuning toward fine-grained relation reasoning, ultra-high-resolution perception, and active visual interaction.

Among these datasets, LHRS-Bot[29] introduces LHRS-Align and LHRS-Instruct. LHRS-Align contains 1.15M image-text pairs collected from Google Earth and OSM, while LHRS-Instruct provides RS-oriented instruction data for image understanding and complex visual reasoning. SkySenseGPT [31] constructs FIT-RS with more than 1.8M samples based on scene graph data, covering both simple and complex comprehension tasks. It further incorporates additional training data from scene classification, VQA, and object detection datasets. RS-GPT4V [61] integrates captioning, VQA, visual grounding, region-level captioning, multi-turn conversation, and description data, using GPT-4V to generate instruction-following samples.

More recent datasets show a clear trend toward scale, task diversity, and high-resolution understanding. Falcon [34] builds a large-scale SFT dataset with approximately 78M samples from 5.6M RS images and supports 14 tasks across image-, region-, and pixel-level understanding. Geo-R1 [38] focuses on few-shot geospatial referring expression understanding and constructs task-specific datasets for REC, OVD, and GRES. ZoomEarth [36] introduces LRS-GRO for ultra-high-resolution RS VQA with active cropping–zooming annotations, while GeoLLaVA-8K [35] builds SuperRS-VQA and HighRS-VQA to support 8K-resolution RS image understanding. These developments indicate that RS instruction tuning datasets are evolving from task-specific collections toward large-scale, fine-grained, and high-resolution multimodal corpora.

0.1pt

2.6 Model Architecture and Fine-tuning↩︎

As illustrated in Figure 3, a typical MLLM consists of a modality encoder, an input projector, an LLM backbone, and, in some cases, an output projector or modality generator for multimodal generation. The modality encoder extracts visual or multimodal features, the input projector maps them into the language embedding space, and the LLM backbone performs instruction following, reasoning, and response generation. Early RS-MLLMs were mainly built upon LLaMA and Vicuna-based architectures due to their accessibility and compatibility with LLaVA-style visual instruction tuning [98]. More recently, stronger general-purpose MLLM backbones, such as Qwen-VL, Qwen2.5VL, Qwen3-VL, InternVL, and GLM-based models, have been increasingly adopted because of their improved multilingual understanding, visual reasoning, and high-resolution perception. This trend is also reflected in recent RS-MLLMs, where Geo-R1 and ZoomEarth are initialized from Qwen2.5VL, while GeoLLaVA-8K adopts LLaVA-Next for ultra-high-resolution image understanding.

The central challenge in MLLMs is to align heterogeneous modality features with the semantic space of LLMs while preserving their instruction-following and reasoning capabilities. Existing training pipelines usually include multimodal pretraining and instruction tuning. The former aligns visual representations with textual embeddings through the input projector, whereas the latter improves task generalization, interaction ability, and domain-specific reasoning. Most early RS-MLLMs rely on supervised fine-tuning (SFT) or parameter-efficient fine-tuning methods, such as LoRA, to reduce training cost. Recent methods further introduce reinforcement-based optimization to enhance reasoning and grounding. For example, Geo-R1 adopts GRPO-based reinforcement fine-tuning for geospatial referring expression understanding, while ZoomEarth combines SFT and GRPO with cropping–zooming tools for ultra-high-resolution image interpretation.

Table 3 summarizes the architecture and tuning strategies of representative RS-MLLMs. Overall, early models commonly freeze the visual encoder and adapt the system through the input projector and LoRA-based LLM tuning. RSGPT is a lightweight example that only tunes the input projector, whereas H²RSVLM fine-tunes the visual encoder, input projector, and LLM during pretraining. LHRS-Bot adopts a more elaborate multi-stage pipeline by combining alignment data, multitask instruction data, detailed descriptions, and selected public reasoning data. SkySenseGPT also incorporates additional public datasets for fine-grained RS instruction tuning. More recent RS-MLLMs show a clear shift from simple projector-level adaptation toward stronger foundation backbones and task-oriented training paradigms. Falcon performs large-scale SFT across multiple RS tasks, Geo-R1 introduces reinforcement fine-tuning for geospatial reasoning, ZoomEarth integrates active perception with cropping–zooming tools, and GeoLLaVA-8K improves high-resolution image understanding through token pruning and visual token selection. These developments indicate that RS-MLLMs are moving toward stronger backbones, high-resolution perception, reasoning-oriented training, and tool-augmented interaction.

0.2pt

2.7 Comments on RS-MLLMs↩︎

Limited standardized evaluation. Table 5 summarizes the quantitative evaluation protocols and datasets used in representative RS-MLLMs. Existing studies have covered diverse tasks, including scene classification, visual question answering, visual grounding, image captioning, object detection, segmentation, region-level captioning, and change detection. However, their evaluation settings remain inconsistent. Many models are evaluated on held-out splits of constructed instruction datasets or benchmarks closely related to their training data, while independent ZS or out-of-domain evaluations are still limited. This makes it difficult to distinguish whether the reported gains arise from general multimodal reasoning, RS-specific knowledge, or dataset/task overlap. Moreover, complex abilities such as multi-turn interaction, grounded description, active perception, and high-resolution reasoning are often demonstrated qualitatively but lack standardized quantitative protocols. These limitations motivate a unified evaluation of RS-MLLMs and general CV-MLLMs.

Template-dependent task execution. Many RS-MLLMs rely on task-specific identifiers or predefined templates, such as “grounding”, “identify”, “refer”, or “[detection]”, to trigger specific outputs. Although such designs can improve task execution under predefined settings, they may also make model behavior sensitive to prompt formats. Recent models have started to mitigate this issue. For example, Falcon introduces dynamic prompt training by sampling semantically equivalent instructions from a predefined prompt pool, aiming to reduce reliance on task-specific tokens [34]. Nevertheless, its training still depends on standardized task templates and structured output formats. This suggests that current RS-MLLMs have not fully moved beyond template-defined task execution, which may limit flexible instruction following in open-ended RSISU scenarios.

0.1pt

3 Evaluation of MLLMs for RSISU↩︎

MLLMs’ capacities and enabled tasks. The capabilities of MLLMs can be classified in various ways. One approach involves a hierarchical taxonomy of human-like capabilities, with perception and reasoning as the top-level categories. Perception is further divided into image-wise, instance-wise, and pixel-wise capabilities, while reasoning encompasses logical and attribute-based reasoning [24].

Alternatively, MLLMs can be categorized based on the format of inputs and outputs, including languages, numbers, coordinates, and images. For answers that consist solely of text, preliminary tasks include VQA and captioning, which can be further classified into scene-level, region-level (focusing on specific areas indicated by text or bounding boxes, such as in referring expression generation), sensor-level (e.g., SAR-level for SAR images), and language-level (e.g., supporting languages like Chinese in addition to English). Questions can also be categorized into types such as counting, geospatial localization, or physical property assessment. For tasks where text alone is insufficient, grounding tasks require bounding boxes in the answers. Visual grounding is divided into two specific tasks: phrase localization, which involves identifying and annotating all objects mentioned in a sentence and referring expression comprehension (REC), which focuses on locating a single object indicated by a language description. Additional tasks that fall into this category include grounded IC, grounded question answering, referring expression segmentation, open-vocabulary detection, and open-vocabulary segmentation.

This section will provide a balanced comparison between RS-MLLMs and representative CV-MLLMs (as listed in Table 7) on common RSISU tasks. Although results reported in existing literature are included as references, our discussion primarily relies on the results obtained in this study, as indicated in the tables, to ensure consistency in experimental settings and evaluation protocols. The accuracies reported in this study may differ from those found in the literature, as observed for models like LLaVA-1.5 and GeoChat in Table ¿tbl:tab:sceneC?. These discrepancies can be attributed to variations in the evaluation methodologies and differences in the prompts used. For example, some models include a period at the end of an answer, which can affect accuracy when answers are directly compared to ground truth. Therefore, post-processing is necessary to achieve more reasonable accuracy. Additionally, the inherent ambiguity of language presents challenges in standardizing evaluations across different implementations, as MLLM performance can be influenced by prompt selection. This variability complicates the effort to make strictly fair comparisons between different MLLMs, as they may respond better to specific prompts.

0.1pt

3.1 Evaluation Protocol↩︎

We adopt a standardized evaluation protocol to ensure fair and reproducible comparisons across all models. All methods are evaluated using publicly released checkpoints or API versions under a zero-shot setting, without any task-specific fine-tuning or adaptation.

To account for potential benchmark contamination, Table 6 summarizes benchmark release dates, possible training-data overlap, evaluation settings, and contamination risks. Accordingly, we distinguish strict zero-shot evaluation from no-adaptation inference, benchmark-aware evaluation, and transfer learning. The reported risk levels indicate potential exposure rather than confirmed contamination.

For visual inputs, we follow each model’s native preprocessing pipeline to avoid introducing additional bias. All models are evaluated under consistent task-specific prompt templates with strict output format constraints (e.g., binary answers or multiple-choice letters), ensuring identical instruction conditions across methods.

For generation, we adopt greedy decoding (\(\text{do\_sample}=\texttt{False}\)) for all models with task-specific maximum token lengths. Model outputs are post-processed using a unified deterministic parsing pipeline, including reasoning-tag removal and task-specific normalization for classification, VQA, multiple-choice, and visual grounding tasks.

We evaluate five representative tasks, including scene classification, visual question answering, multi-label recognition, multiple-choice reasoning, and visual grounding, across eleven benchmark datasets. Classification tasks are evaluated using Overall Accuracy, while VQA and multiple-choice tasks report accuracy. Visual grounding is evaluated using mean Intersection-over-Union (mIoU) and Acc@0.5/0.7.

3.2 Performance on task-specific RS datasets↩︎

3.2.1 Scene Classification↩︎

Table ¿tbl:tab:sceneC? presents the comparative remote sensing image scene classificaition results for both single-label and multi-label tasks, which are commonly evaluated in current RS-MLLM studies. On relatively simple datasets, such as AID [80], WHU-RS19 [81] and EuroSAT [82], several literature-reported RS-MLLMs achieve highly competitive performance. In particular, LHRS-Bot, SkySenseGPT, and H²RSVLM obtain the best or near-best results, even surpassing the proprietary GPT-4o on these two benchmarks. This demonstrates the effectiveness of remote-sensing-specific fine-tuning for scene-level recognition. However, this advantage becomes less evident under our unified evaluation setting. On more challenging benchmarks, including EuroSAT with 10 m GSD imagery and AID-multi with multi-label annotations, CV-MLLMs, including both open-source and closed-source models, generally outperform existing RS-MLLMs. These results suggest that current RS-MLLMs still face limitations when handling lower-resolution imagery and complex multi-label scene understanding.

0.1pt

The results also reveal clear differences in dataset difficulty. WHU-RS19 appears to be the least challenging benchmark, partly because it contains only 19 scene categories and consists of very high-resolution images. AID is also composed of VHR images but includes a larger number of categories, making it relatively more difficult. In contrast, EuroSAT is based on Sentinel-2 imagery with a 10 m GSD, which differs substantially from both natural images commonly used to train general CV-MLLMs and the VHR imagery frequently adopted in RS-MLLM training. This domain and resolution gap makes EuroSAT particularly challenging for existing RS-oriented models.

Overall, the findings highlight the strong generalization ability of recent MLLMs in the remote sensing domain. Among the closed-source models, GPT-4o achieves the best overall performance, especially on EuroSAT and AID-multi. Nevertheless, it is not consistently the best model across all datasets, as InternVL3.5 and GLM-4.6V-Flash achieve higher accuracy on WHU-RS19 under our evaluation. Among the open-source CV-MLLMs evaluated in this study, InternVL3.5, GLM-4.6V-Flash, and Qwen3-VL show the most competitive performance, clearly outperforming earlier models such as Qwen2-VL, LLaVA-1.6, Tiny-LLaVA, CogVLM_chat, and Qwen-VL-Chat on most benchmarks. In contrast, GeoLLaVA-8K performs poorly across most datasets, suggesting that its SFT process may lead to limited generalization, with the model overfitting specific instruction patterns and failing to robustly answer diverse scene classification queries.

3.2.2 Visual Question Answering↩︎

Table ¿tbl:tab:rsvqa? presents the VQA performance of RS-MLLMs and CV-MLLMs on Test Set 2 of RSVQA-HR [84], a widely used benchmark in the literature. Following common evaluation practice, area and count questions are excluded. For RSVQA-HR, we follow a controlled evaluation protocol by selecting two representative question types, i.e., Presence and Comparison, which are the most commonly studied reasoning categories in prior RS-VQA literature and directly reflect object existence reasoning and relational reasoning ability in remote sensing images. In this study, the results are obtained from a random sample of 528 Presence-type and 528 Comparison-type questions. Overall, RS-MLLMs show clear advantages over general CV-MLLMs on this benchmark, especially on the more challenging Comparison-type questions. Among them, Falcon achieves the best overall performance, while RS-LLaVA also obtains strong results. Literature-reported models, such as SkySenseGPT and H²RSVLM, also show competitive performance, further indicating the effectiveness of remote-sensing-specific instruction tuning for RSVQA-HR.

Among the eight CV-MLLMs evaluated in this study, InternVL3.5 achieves the highest average accuracy, followed by Qwen2-VL, TinyLLaVA, and Qwen3-VL. Although GPT-4o shows stable performance across the two question types, it does not achieve the best results on this task and is surpassed by several open-source CV-MLLMs. These results suggest that recent open-source general-purpose MLLMs have achieved improved transferability to remote sensing VQA, but they still lag behind the strongest RS-oriented models. The larger performance gaps on Comparison questions suggest that fine-grained object comparison and spatial reasoning remain key challenges for general-purpose MLLMs in high-resolution remote sensing imagery.

3.2.3 Visual Grounding↩︎

Table 9 presents the visual grounding results on three representative remote sensing grounding benchmarks, including VRSBench [87], DIOR-RSVG [86], and RSVG [85]. Following common grounding evaluation protocols, model performance is measured using Acc@0.5, Acc@0.7, and mIoU, where Acc@0.5 and Acc@0.7 indicate whether the predicted bounding box overlaps with the ground-truth box by more than 0.5 and 0.7 Intersection-over-Union (IoU), respectively.

Overall, RS-MLLMs exhibit stronger grounding performance than general CV-MLLMs, especially on VRSBench and DIOR-RSVG. This advantage may be partly attributed to remote-sensing-specific instruction tuning and the possible inclusion of related grounding datasets, such as RSVG or DIOR-RSVG, in the training data of some RS-oriented models. Among all evaluated models, RSThinker achieves the best performance across all three datasets and all evaluation metrics, demonstrating a clear advantage in precise object localization. In particular, its substantial gains under the stricter Acc@0.7 metric suggest that it can produce more accurate bounding boxes rather than merely identifying approximate target regions.

In addition, reasoning-oriented VLMs, such as GLM-4.1V-Thinking, also show competitive performance, indicating that explicit reasoning ability is beneficial for visual grounding tasks that require understanding referring expressions, spatial relationships, and object locations. In contrast, closed-source commercial VLMs and most general CV-MLLMs show limited localization precision, especially under the Acc@0.7 metric. Moreover, Figure 6 further illustrates the promising performance of typical MLLMs. These results suggest that fine-grained visual grounding in remote sensing remains a challenging task for current general-purpose MLLMs.

3.3 Performance on MLLM-specific RS Benchmarks↩︎

3.3.1 LHRS-Bench Benchmark↩︎

LHRS-Bench is a comprehensive benchmark designed for evaluating RS-MLLMs. It contains 690 single-choice questions organized into five top-level dimensions: Recognition, Imagery, Spatial-awareness, Quantity, and Reasoning. Recognition focuses on identifying geographical objects and their attributes, such as color and shape. Imagery evaluates the understanding of image resolution and modality. Spatial-awareness assesses the ability to reason about object locations and distances, while Quantity and Reasoning evaluate counting ability and domain-specific visual reasoning. These dimensions are further divided into 11 sub-dimensions. To reduce the risk of data leakage, LHRS-Bench uses 108 images collected exclusively from Google Earth rather than public remote sensing datasets. Its single-choice format, with 2–4 candidate answers for each question, enables quantitative and objective evaluation.

Table 8 presents the evaluation results on LHRS-Bench. Overall, recent general-purpose MLLMs demonstrate strong performance on this benchmark. GPT-4o achieves the best average accuracy among all evaluated models. Among open-source CV-MLLMs, GLM-4.6V-Flash obtains the highest average accuracy, followed by Qwen3-VL and Qwen2-VL. These models substantially outperform early CV-MLLMs, such as MiniGPTv2, InstructBLIP, and LLaVA-1.5, indicating the rapid progress of general visual-language models and their strong transferability to remote sensing understanding.

In contrast, existing RS-MLLMs do not consistently outperform general CV-MLLMs on LHRS-Bench. Although ZoomEarth and Geo-R1 achieve competitive performance, several RS-oriented models, such as GeoChat, RS-LLaVA, and GeoLLaVA-8K, remain clearly behind recent general-purpose MLLMs. This suggests that remote-sensing-specific instruction tuning alone is insufficient to guarantee strong generalization across diverse question types. The category-wise results further show clear performance variation across different task types. Most models perform relatively well on Identity, Shape, and Reasoning questions, but still struggle with Orientation, Modality, and Resolution. These results indicate that LHRS-Bench requires not only remote sensing domain knowledge, but also robust visual perception, spatial reasoning, and instruction-following ability.

3.3.2 FIT-RSRC Benchmark↩︎

The FIT-RSRC dataset is designed to evaluate fine-grained relation comprehension in remote sensing scenes using multiple-choice questions. It contains four question types: Subject, Object, Relation, and Existence, each accounting for 25% of the dataset. Negative and unanswerable cases are also included to assess model robustness in ambiguous or non-existent relation scenarios. Most questions provide four candidate answers, while a smaller portion provides five options. To improve the quality of distractors, the incorrect options are carefully checked against a curated commonsense word list, preventing models from identifying the answer purely through textual priors.

Table ¿tbl:tab:fitrc? presents the comparative performance of RS-MLLMs and CV-MLLMs on FIT-RSRC. Different from some traditional remote sensing tasks, the strongest results on this benchmark are achieved by recent general-purpose CV-MLLMs. Qwen3-VL obtains the highest average accuracy among all models, followed by GLM-4.6V-Flash. Both models outperform GPT-4o and all evaluated RS-MLLMs under the reported settings. This indicates that FIT-RSRC requires strong fine-grained visual perception, compositional reasoning, and instruction-following ability, rather than relying solely on remote sensing domain-specific tuning.

Among RS-MLLMs, Geo-R1 achieves the best average accuracy in this study, but it still lags behind the strongest general-purpose CV-MLLMs. SkySenseGPT also achieves competitive performance, especially on Object, Relation, and Existence questions. However, its result is obtained under a transfer learning setting, making it not directly comparable with strict zero-shot evaluations. The question-type analysis shows that Existence is generally the easiest category for most models, while Object and Relation remain more challenging because they require fine-grained object recognition and inter-object relationship understanding. Subject questions are handled well by several recent MLLMs, particularly GLM-4.6V-Flash and Qwen-3VL, but earlier models and some RS-MLLMs still show limited performance. These results suggest that current RS-MLLMs still need stronger compositional reasoning and broader instruction generalization to perform well on fine-grained remote sensing relation comprehension.

3.3.3 XLRS-Bench Benchmark↩︎

XLRS-Bench further complements LHRS-Bench and FIT-RSRC by evaluating MLLMs in ultra-high-resolution remote sensing scenarios. At the L-3 task level, it covers diverse perception and reasoning abilities, including counting, land-use classification, object recognition, multi-scale perception, spatial relationship understanding, anomaly detection, route planning, and change reasoning. In this study, the tested models are evaluated under a zero-shot inference protocol, where no additional fine-tuning is performed on XLRS-Bench. Nevertheless, since some results are collected from public reports or models released after the benchmark became available, they should be interpreted as benchmark-aware evaluations rather than strictly contamination-free zero-shot results. As shown in Table 10, RS-oriented models show strong competitiveness on this benchmark, with ScaleEarth achieving the best average performance, followed by GeoEyes and GeoLLaVA-8K. This differs from LHRS-Bench and FIT-RSRC, where recent general-purpose CV-MLLMs often perform strongly. These results suggest that domain-specific adaptation remains important for ultra-high-resolution remote sensing understanding, especially when models need to integrate fine-grained perception, spatial reasoning, regional semantics, and remote-sensing-specific knowledge.

3.4 Complex RSISU Cases↩︎

Figures 4 and 5 show two representative failure cases of current MLLMs on complex RSISU tasks requiring spatial reasoning. In Figure 4, the model needs to infer the direction of sunlight based on the shadows cast by buildings. This is relatively straightforward for humans, as the illumination direction can be inferred as opposite to the shadow direction. However, even strong models such as Qwen3-VL and Geo-R1 fail on this case. Qwen3-VL predicts the direction as top-left, while Geo-R1 answers northwest, both of which are incorrect. This suggests that current MLLMs still lack robust shadow-based spatial reasoning ability in remote sensing scenes. In addressing the question in Figure 5, the model needs to understand the runway layout, identify the airplanes, and correctly interpret the spatial term "right". However, more than half of the tested MLLMs fail to provide the correct answer, indicating that fine-grained counting and spatial relation understanding remain challenging for current MLLMs.

Figure 10 presents representative RSISU examples that require distinguishing among true-color optical images, false-color optical images, and SAR images, as well as performing multi-view image analysis and multi-temporal change detection. When these questions are posed to state-of-the-art MLLMs, the responses are generally unsatisfactory. In particular, most models struggle to handle multiple images as input, which limits their ability to perform cross-view comparison and temporal reasoning. Since such tasks are common in practical RSISU scenarios, these results reveal a substantial gap between the current capabilities of MLLMs and the requirements of real-world remote sensing applications.

3.0pt

3.5 Comments on Evaluation Results↩︎

The comparative results reveal that the relative advantages of RS-MLLMs and CV-MLLMs are highly task-dependent. On conventional remote sensing scene classification and VQA benchmarks, several RS-MLLMs achieve competitive or even superior performance, especially on datasets whose visual distributions are close to those used during RS-specific instruction tuning. For example, RS-oriented models show clear advantages on RSVQA-HR and visual grounding tasks, where domain-specific object categories, spatial expressions, and grounding annotations are important. However, this advantage becomes less consistent on recently proposed RS-MLLM-specific benchmarks, such as LHRS-Bench and FIT-RSRC. On these datasets, recent general-purpose CV-MLLMs, including GPT-4o, Qwen3-VL, GLM-4.6V-Flash, and Qwen2-VL, often outperform most existing RS-MLLMs. This suggests that current RS-MLLMs may still suffer from limited instruction diversity and insufficient generalization beyond their training distributions.

The progress of general-purpose CV-MLLMs is clearly reflected in their remote sensing performance. Compared with earlier models such as Qwen-VL-Chat, LLaVA-1.5, and MiniGPT-v2, recent models demonstrate substantially stronger visual perception, instruction following, and reasoning abilities. In particular, Qwen3-VL and GLM-4.6V-Flash achieve strong results on LHRS-Bench and FIT-RSRC, while InternVL3.5 also shows competitive performance on RSVQA-HR. These observations indicate that advances in general multimodal foundation models can transfer effectively to remote sensing tasks, even without explicit RS-specific fine-tuning.

The results also show that model size alone does not determine performance. Some relatively compact models, such as TinyLLaVA, outperform several RS-MLLMs on specific benchmarks, while larger or domain-tuned models may still fail on tasks requiring robust spatial reasoning, multi-image understanding, or fine-grained relation comprehension. This indicates that the quality and diversity of instruction data, the alignment strategy, and the reasoning capability of the base model are often more important than parameter scale alone.

Overall, these findings suggest that future RS-MLLM development should not rely solely on domain-specific supervised fine-tuning over narrow task formats. Instead, more attention should be paid to constructing diverse and carefully curated RS instruction datasets, covering multi-resolution imagery, multi-modal observations, multi-view inputs, multi-temporal change analysis, visual grounding, and compositional reasoning. In addition, stronger general-purpose MLLMs can serve as effective backbones or teachers for RS-MLLMs, while remote-sensing-specific training should focus on complementing their weaknesses in spatial reasoning, sensor-aware understanding, geographic knowledge, and precise localization. Such a combination of strong general multimodal capabilities and well-designed RS-specific adaptation is likely to be a more promising direction for building robust and practical RSISU systems.

4 Open Problems and Future Directions↩︎

Although RS-MLLMs have shown promising potential for remote sensing image understanding, the above comparisons reveal that their capabilities remain far from mature. Their performance is highly task-dependent: RS-MLLMs show clear advantages on visual grounding, RSVQA-HR, and XLRS-Bench. In contrast, recent general-purpose CV-MLLMs achieve stronger or comparable performance on LHRS-Bench, FIT-RSRC, and several scene classification settings. These findings indicate several important open problems for future research.

4.1 Reliable and Contamination-aware Evaluation↩︎

Fair and reproducible evaluation is increasingly important for RS-MLLMs. Existing results are drawn from heterogeneous sources, including literature reports, reproduced experiments, and API-based evaluations, which complicates direct comparison. As summarized in Table 6, contamination risks vary across benchmarks and evaluated models. Future studies should therefore report benchmark and model release dates, potential training-data overlap, exact model or API versions, prompts, and evaluation protocols when presenting zero-shot results. Such transparency is essential for distinguishing zero-shot evaluation with a low risk of contamination from benchmark-aware inference and enabling reliable model comparisons.

4.2 Generalization beyond Narrow RS Instruction Tuning↩︎

Current RS-MLLMs do not consistently outperform general CV-MLLMs across all benchmarks. As shown in Fig. 11, narrow instruction tuning limits the instruction-following and generalization capabilities of GeoLLaVA-8K. Although it performs well on ultra-high-resolution RS-VQA, it often produces invalid or inconsistent responses for out-of-distribution tasks such as scene classification and visual grounding, indicating limited adaptability to broader RSISU tasks. This suggests that some RS-specific models may overfit limited instruction formats, visual distributions, or task types. Future work should move beyond narrow supervised fine-tuning and construct more diverse RS instruction datasets. These datasets should cover different sensors, resolutions, geographic regions, object categories, question formats, and reasoning patterns, so that RS-MLLMs can generalize to unseen tasks rather than only performing well on familiar benchmarks.

4.3 Spatial Reasoning and Fine-grained Grounding↩︎

Spatial reasoning remains a major bottleneck for current MLLMs. The failure cases discussed above show that many models struggle with shadow-based sun-direction reasoning, left/right relations, object counting, orientation understanding, distance estimation, and precise visual grounding. These tasks require not only object recognition but also geometric and physical reasoning. Future RS-MLLMs should incorporate geometry-aware supervision, spatial relation verification, and region-level grounding ability to support reliable interpretation of complex remote sensing scenes.

4.4 Improved Performance by Training-free Methodologies↩︎

Developing RS-MLLMs usually requires substantial efforts in domain-specific data collection, annotation, and instruction tuning. However, our quantitative analysis shows that such resource investment does not necessarily lead to consistently superior performance. As discussed in Section 3, several CV-MLLMs achieve competitive performance compared with RS-MLLMs on certain RS tasks, even without explicit adaptation to RS instruction data. This suggests that training-free methodologies, such as prompt engineering, in-context learning, reasoning-oriented prompting, and tool-assisted inference, remain promising but insufficiently explored directions for improving RS-MLLMs. Previous studies have shown that few-shot learning and Chain-of-Thought prompting can enhance the reasoning and generalization abilities of large models. In our experiments, as shown in Figure 7, adding a one-shot example to the prompt increases the classification accuracy on EuroSAT from 56.7% to 72.4%, indicating the effectiveness of training-free adaptation for RS scene understanding.

We further observe that prompt formulation can substantially affect model behavior, particularly for tasks requiring fine-grained grounding or structured outputs. For example, the prompt “<phrase>A chimney on the right.</phrase>” enables KOSMOS-2 to produce accurate referring expression comprehension results, whereas the plain prompt “A chimney on the right.” fails to activate the desired grounding capability. Minor formatting changes can also influence the output; for instance, removing the period in “<phrase>A chimney on the right</phrase>” leads to inferior results. These observations indicate that prompt design is an important factor affecting the reliability of MLLMs in RS applications, especially in multi-class classification and visual grounding tasks, where task instructions, output formats, and grounding markers directly influence predictions, as illustrated in Figure 8 and Figure 9. Nevertheless, the effectiveness of these strategies is model- and task-dependent. Therefore, future RS-MLLM studies should systematically evaluate task-aware and model-specific training-free strategies before relying on costly domain-specific instruction tuning.

4.5 Multi-modal, Multi-image, and Ultra-high-resolution Understanding↩︎

Real-world RSISU often involves heterogeneous inputs, including true-color optical images, false-color composites, SAR, infrared, multispectral, hyperspectral, multi-view images, and multi-temporal observations. Although recent efforts such as Earth-OneVision [102] and RemoteShield [39] have explored multimodal MLLMs for remote sensing, most existing RS-MLLMs still rely primarily on single-modality inputs, particularly RGB imagery, leaving multi-source RS data insufficiently exploited. In addition, ultra-high-resolution RS images contain rich local details but are difficult to process due to resolution and context-length limitations. Future models should support sensor-aware encoding, multi-image comparison, temporal reasoning, adaptive region selection, and hierarchical high-resolution understanding.

4.6 Efficient and Tool-augmented RSISU Systems↩︎

Building and deploying RS-MLLMs require substantial computational resources, especially for high-resolution inputs and large-scale instruction tuning. Therefore, efficiency should be a key future direction, including lightweight architectures, visual token compression, parameter-efficient tuning, and training-free adaptation methods such as prompt engineering and few-shot prompting. In addition, RS agents that combine MLLMs with specialized tools, such as object detectors, segmentation models, retrieval systems, and GIS software, offer a practical path toward reliable RSISU. As shown in Table ¿tbl:tab:rs-agent?, integrating BLIP [7] enables more comprehensive image caption generation, while SAM [103] provides fine-grained spatial information that facilitates grounding queries in MLLMs. In addition, the directions discussed for RS agents in Sec 2.4, such as reliable tool selection, long-horizon reasoning, trajectory verification, multi-agent collaboration, and domain-specific geospatial workflows, remain important topics for further exploration.

5 Conclusions↩︎

This survey provides a systematic review and diagnostic evaluation of MLLMs for remote sensing image scene understanding. Beyond summarizing model architectures, training data, and adaptation strategies, our analysis supports four main conclusions about the current development of RS-MLLMs.

First, the observed advantages of RS-MLLMs are localized rather than universal. RS-oriented models perform strongly on several domain-specific tasks, particularly remote sensing VQA, visual grounding, and ultra-high-resolution image understanding. These advantages are most evident when the evaluation data and task formats are closely related to those used for RS instruction tuning. However, they do not yet translate into consistent superiority across diverse RSISU tasks. Current RS-specific adaptation therefore improves selected capabilities but has not established general remote sensing understanding.

Second, general foundation capabilities increasingly determine benchmark performance. Recent general-purpose CV-MLLMs achieve competitive or superior results on several RS benchmarks without explicit RS-specific fine-tuning. Their performance is supported by strong visual perception, instruction following, and compositional reasoning. These capabilities can be more influential than model size or narrow domain-specific training. General-purpose MLLMs should therefore be considered important backbones or teachers for future RS-MLLMs rather than merely external baselines.

Third, the evaluation protocol has become a central bottleneck. Existing comparisons often combine literature-reported results, reproduced experiments, API-based evaluations, and transfer-learning results. Potential benchmark contamination further complicates their interpretation. Future studies should clearly distinguish strict contamination-free zero-shot evaluation, no-adaptation inference, benchmark-aware evaluation, and transfer learning. They should also report model versions, benchmark release dates, training-data overlap, prompts, and output-processing rules. Without such controls, the benefits of RS-specific adaptation may be overestimated.

Finally, the next frontier lies in sensor-aware and verifiable remote sensing intelligence. Future research should move beyond repeatedly expanding conventional captioning and VQA datasets. Greater attention is needed for multisensor, multi-image, and multitemporal understanding. Models must also improve geometric reasoning, precise grounding, geographic knowledge, and robustness across task formats. Tool use, external knowledge, and agent-based verification provide promising ways to make model outputs more traceable and reliable.

The central conclusion is therefore clear: domain specificity alone is insufficient for building general and practical RS intelligence. Future progress will depend on combining strong general-purpose multimodal foundations with targeted RS adaptation. Such adaptation should complement the weaknesses of general models through sensor-aware perception, spatially grounded reasoning, and geographic knowledge.

Acknowledgments↩︎

This work was supported in part by the National Natural Science Foundation of China under Grant 62525108, Grant 62371185, Grant 62006241, and Grant 42201513, in part by the National Key Research and Development Program of China under Grant 2021 YFA0715203, in part by the Science and Technology Innovation Program of Hunan Province under Grant 2024RC1030 and Grant 2023RC3124, in part by the Project of Yuelushan Center for Industrial Innovation under Grant 2025YCII0202, in part by the China Postdoctoral Science Foundation under Grant 2022M723902 and Grant 2023T160789.

Declaration of Interest Statement↩︎

The authors declare no competing financial interests or personal relationships that could have influenced the work reported in this paper.

References↩︎