1 Introduction↩︎

Humans possess a remarkable ability to interpret high-dimensional, uncompressed visual input and abstract its underlying structure, enabling the manipulation of underlying concepts as symbolic representations with notable efficiency [1], [2]. As a result, humans can effortlessly locate a target object in a cluttered room, determine whether a cup will fit inside a drawer, or predict the direction in which a stack of objects might fall. This cognitive capability, known as visual reasoning, is widely regarded as a concentrated embodiment of human intelligence, forming the basis for concept formation, world understanding, and interaction with the environment [2]–[5]. At its core, visual reasoning is inherently compositional: it requires recognizing objects and attributes, grounding them in visual evidence, modeling relations among them, and recombining these elements to support new inferences [4]–[6].

In pursuit of human-level intelligence, a growing body of research has sought to replicate visual reasoning capabilities in machines [1], [7]–[9]. Recent advances in vision-language pretraining and large multimodal models have greatly expanded the scope of machine visual reasoning, enabling end-to-end systems to map visual and textual inputs directly to answers across natural images and embodied environments [10]–[14]. These systems have shown impressive results in general-purpose multimodal understanding [15]–[17] and have been increasingly adopted across real-world applications [10], [18]–[22].

Despite the promising performance of monolithic approaches in general multimodal understanding, significant challenges remain, particularly when viewed from the perspective of the inherently compositional and multi-step nature of human visual reasoning [14], [23]–[28]. Concretely, we identify the following key limitations of monolithic approaches:

• Bias-driven shortcuts. Instead of performing grounded inference, monolithic models often exploit spurious correlations and linguistic priors to produce plausible but incorrect answers [23], [24], undermining their generalization to challenging or novel scenarios.

• Poor scaling with reasoning complexity. Scaling up data and compute does not yield proportional gains when tasks demand multi-hop reasoning, spatial understanding, or precise grounding [29]–[33].

• Weak compositional generalization. Holistic models struggle to recombine objects, attributes, and relations in novel configurations, limiting their alignment with human-like visual reasoning [4]–[6], [34]–[36]. Monolithic models, by contrast, treat the input holistically and lack mechanisms for explicit decomposition and relational reasoning.

These limitations highlight the need for modular, interpretable, and compositionally aligned approaches to visual reasoning.

Motivated by these cognitive insights and the limitations of monolithic models, recent research has increasingly explored Compositional Visual Reasoning (CVR), a paradigm that decomposes visual tasks into grounded, interpretable reasoning steps guided by visual perception. Rather than relying on direct end-to-end prediction, CVR aims to make intermediate evidence and inference processes explicit, enabling models to reason over objects, attributes, relations, actions, and external knowledge in a structured manner [4], [29]. As illustrated in Figure 1, this survey focuses on methods that expose or construct such intermediate reasoning processes before answer generation.

Given its rapid progress since 2023, compositional visual reasoning has emerged as a central paradigm in the study of visual intelligence. However, its development trajectory, methodological foundations, and future directions remain insufficiently examined. Existing surveys in visual reasoning have primarily focused on conventional or monolithic approaches [37]–[39], and thus fail to capture the emerging trend of compositional visual reasoning, which has quickly become a core research direction. Meanwhile, surveys focused on multimodal large language models and reasoning tend to emphasize general-purpose reasoning [31], [40], neurosymbolic frameworks [41], abstract pattern recognition [42], or agent-based architectures [31], [43]. Although related surveys cover aspects such as planning, perception, and symbolic abstraction, they do not systematically examine CVR as a distinct paradigm. As shown in Table 1, a dedicated review of this area is still lacking.

To address this gap, we primarily focus on image-centered compositional visual reasoning, while discussing 3D, multi-view, embodied, and world-model-based methods when they directly extend or illuminate image-based CVR mechanisms. We focus on methods published between January 2023 and June 2026 that introduce explicit intermediate reasoning steps before answer generation, including LLM-guided inference, tool-integrated workflows, chain-of-thought-based visual reasoning, and agentic VLM architectures. Video reasoning and 3D reasoning are not treated as standalone survey targets, but are discussed when they directly inform the methodological development of image-based CVR. The main contributions of this survey are fourfold.

1. Systematic motivation for CVR. We clarify why CVR is needed for modern multimodal systems, focusing on grounding, compositional generalization, interpretability, robustness, and efficiency.

2. A stage-wise taxonomy of CVR methods. We propose a five-stage taxonomy of large-model-enhanced CVR methods, spanning prompt-based reasoning, tool-augmented workflows, grounded chain-of-thought, and agentic VLMs.

3. Comprehensive review of benchmarks and evaluation. We summarize representative datasets and evaluation settings used in CVR research, highlighting their task types, visual inputs, reasoning targets, and evaluation metrics.

4. Open challenges and future directions. We synthesize key bottlenecks and outline future directions toward grounded, interpretable, robust, and compositionally generalizable visual reasoning.

The remainder of this survey is organized as follows. Sec. 2 introduces key definitions, including visual reasoning, monolithic visual reasoning, and compositional visual reasoning. Sec. 3 discusses why CVR is needed, covering cognitive alignment, semantic and relational understanding, robustness, interpretability, modular reuse, bias mitigation, and data efficiency. Sec. 4 presents our multi-stage taxonomy of CVR methods, from prompt-enhanced LLM-centric pipelines to tool-augmented systems, CoT-based VLMs, and unified agentic VLMs. Finally, we summarize key challenges, and discuss future directions in Sec. 6.

2 Background↩︎

In this section, we provide an overview of the key concepts and terminology related to compositional visual reasoning, offering a clearer understanding of the fundamental aspects of this field.

2.1 Visual Reasoning↩︎

Visual reasoning is a cognitive process that enables the interpretation and analysis of relationships among entities within a visual scene to support decision-making and problem-solving [15], [26], [42], [49]. Rather than being limited to low-level perception, visual reasoning involves the integration of high-level information from multiple modalities (, vision and language) to facilitate deeper semantic understanding [39], [44]. This multimodal reasoning underlies tasks such as visual question answering, entailment, and grounding [50], [51], where accurate comprehension and logical analysis of visual content are critical.

In this survey, we focus on visual reasoning tasks that involve interpreting visual inputs (, images) paired with textual queries or prompts to produce structured outputs. Formally, we denote the visual input as \(v\), the query or prompt as \(q\), and the answer as \(y\). A visual reasoning system is a function \(\mathcal{M} : (v, q) \mapsto y\), which varies depending on the reasoning approach, monolithic or compositional. The form of \(y\) also varies depending on the task and may include natural language responses, discrete answer selections (, multiple choice), bounding boxes, or segmentation masks. Representative tasks include, but are not limited to, general visual question answering (, “What color is the car?"), visual grounding (,”Find the chair next to the window"), relational scene graph reasoning (“Is the cat under the table?").

At its core, visual reasoning comprises two essential components: visual perception, which involves extracting visual information such as objects, attributes, and relations [52], [53], and logical reasoning, which applies structured reasoning to derive conclusions based on the perceived information [54], [55]. While intuitive for humans, this capability remains a major challenge for artificial intelligence systems due to the requirement for high-level abstraction and generalization.

2.2 Monolithic Visual Reasoning↩︎

Monolithic visual reasoning methods are a class of neural network models \(\mathcal{M}\) that directly map visual input \(v\) and textual query \(q\) to an output answer \(y\), where \(\mathcal{M}\) commonly directly encodes both modalities into a joint representation and predicts the answer without exposing intermediate steps. These architectures do not explicitly represent reasoning steps, performing reasoning implicitly within end-to-end pipelines [1], [34]. These models typically extract visual features from the entire image \(v\) and combine them with language embeddings using co-attention or joint encoding to perform implicit multimodal reasoning [10], [46], [56], [57].

Early models such as MFB [56], and MCAN [58] used convolutional features and cross-modal attention for direct answer prediction. Recent extensions include dual-encoder contrastive models (, ViLBERT [59], CLIP [60], BLIP2 [12]), single-Transformer backbones (, UNITER [61], Flamingo [62]), and vision-encoder-to-LLM pipelines (, LLaVA [10], MiniGPT-4 [63], InstructBLIP [13], Qwen-VL [11]).

2.3 Compositional Visual Reasoning↩︎

Compositional reasoning approaches map an input pair \((v, q)\) to an answer \(y\) via an intermediate structured representation, for example, a sequence of \(n\) steps \(S = \{s_1, s_2, \dots, s_n\}\), where the decomposition is either inferred dynamically or predefined based on the query structure. Here, \(S\) is used as an abstract representation of the reasoning trajectory rather than a fixed implementation form. Each step \(s_i\) may be instantiated as a symbolic operation, an LLM-generated instruction, a multimodal model invocation, or a call to an external visual tool, depending on the framework. When grounded to the visual input \(v\), a step may involve finer-grained operations such as object identification (\(o_i\)), attribute inference (\(a_i\)), and relation resolution (\(r(o_i, o_j)\)). These grounding processes may encompass visual perception capabilities such as object detection, attribute recognition, and depth estimation accordingly. The steps \(S\) can then be executed sequentially or composed hierarchically into a symbolic program, ultimately enabling the model to produce the answer \(y\).

Compositional visual reasoning is a specialized form of visual reasoning that emphasizes such structured decomposition and recombination of visual elements to solve complex tasks [64]. It is grounded in the principle of compositionality, which posits that “the meaning of the whole is a function of the meanings of its parts” [65]. Rather than treating a scene as a monolithic input, compositional reasoning explicitly identifies and manipulates objects, attributes, and their interrelations to construct flexible reasoning pathways [1], [9], [34].

A hallmark of compositional visual reasoning is its capacity for systematic generalization: the ability to infer novel combinations of familiar elements without direct exposure to those configurations during training [4], [34]. Concretely, this generalization refers to executing reasoning over the combinatorial product space of primitive visual elements, such as objects \(o\), attributes \(a\), and relations \(r(o_i, o_j)\), as well as adapting to different orderings of steps \(S\). This mirrors human-like cognitive flexibility and is central to building AI systems that are interpretable, modular, and capable of robust generalization across diverse multimodal tasks [29], [30].

3 Why Compositional Visual Reasoning?↩︎

In this section, we outline the core motivations for adopting compositional visual reasoning, emphasizing its advantages over traditional monolithic models. Specifically, we examine six key perspectives: cognitive alignment with human reasoning; semantic and relational understanding; generalization and robustness; transparency, interpretability, and modular reuse; reducing language biases and hallucinations; and lower data requirements and improved efficiency.

3.1 Cognitive Alignment with Human Reasoning↩︎

Compositional visual reasoning (CVR) is inspired by the systematic and modular way humans approach visual problems [6], [64], [66]–[69]. Humans naturally break down complex scenes into interpretable components (, objects, attributes, and relationships) and apply sequential reasoning and contextual adaptation [1], [29], [70], [71]. This process reflects our ability to manipulate abstract symbols and relational concepts extracted from visual inputs.

Moreover, the human capacity for compositional reasoning, rapid concept formation, and relational generalization is particularly remarkable [9], [34], [69], [72]. These abilities, often associated with fluid intelligence and non-verbal reasoning, enable flexible understanding across novel contexts and support exceptional sample efficiency when learning new visual concepts. Therefore, a key desideratum for CVR models is to mirror core cognitive strategies, such as task decomposition, focused attention, iterative refinement, and the use of external tools to isolate critical visual details [73], [74].

3.2 Semantic and Relational Understanding↩︎

Traditional monolithic models often rely on superficial correlations and pattern matching, which limits their ability to capture deep semantic structures or precise object-centric relationships. In contrast, compositional visual reasoning enables explicit modeling of such relationships through structured representations like scene graphs or deductive reasoning within the language space [26], [41], [75]–[78]. This allows systems to reason about spatial configurations, semantic relations, and attribute interactions in a principled and interpretable manner [41], [45]. Such capabilities are essential for abstract scene understanding tasks, including counting, identifying intersections, and comparing object properties [40]. Moreover, compositional reasoning facilitates multimodal understanding by bridging vision and language, enabling accurate interpretation of complex visual elements in diagrams and charts [31].

3.3 Generalization and Robustness↩︎

Monolithic visual reasoning models are often brittle and prone to bias, with poor performance on out-of-distribution or few-shot tasks due to shortcut learning [5], [79], [80]. Compositional approaches, by contrast, promote systematic generalization—the ability to solve novel tasks by recombining known elements, such as objects, attributes, or relations [4], [81]. For example, when structured models learn to ground visual concepts in a scene (, “monitor” and “desk”) and resolve relations (, “on top of”), they can generalize to identify any object-relation pair (, monitor on desk), without having seen that specific combination during training. This ability reduces dependence on dataset-specific biases and enhances robustness across domains. Benchmarks like CLEVR [5] and VCR [66] have demonstrated the necessity of compositional reasoning to prevent models from exploiting statistical priors and encourage faithful reasoning [1], [70].

3.4 Transparency, Interpretability, and Modular Reuse↩︎

Compositional visual reasoning enables the generation of intermediate outputs (, bounding boxes, scene graphs, textual rationales) that expose the model’s decision path, improving interpretability and allowing targeted debugging [67], [82]. This ability is essential for building trustworthy and reliable systems, especially in domains where high precision is required. In contrast to black-box architectures, compositional systems are modular: components can be reused across tasks, reducing the need to retrain models from scratch and enabling plug-and-play integration of vision experts [83]–[85]. As individual components improve, so does the full compositional framework. This flexibility also leads to efficiency gains in inference and model adaptation.

3.5 Reducing Language Biases and Hallucinations↩︎

While generalization focuses on performance across diverse input distributions, another critical challenge for VLMs lies in the generation of inaccurate or unsupported content. Monolithic methods, especially those trained to directly map visual inputs to answers or rationales, are highly susceptible to hallucinations. These models might generate responses that appear plausible but are actually incorrect or ungrounded, particularly when they depend too heavily on linguistic priors [86]–[88]. For instance, when asked about the color of a green banana, a monolithic model might incorrectly answer “yellow." This issue stems from shortcut learning, where models exploit statistical biases in the training data rather than grounding their reasoning in visual evidence [5], [79], [89], [90]. In contrast, compositional reasoning mitigates this by enforcing explicit grounding: intermediate reasoning steps, such as object detection, relation extraction, or visual token selection, compel the system to anchor its inferences in actual visual content. This not only improves factual consistency but also reduces reliance on spurious correlations and enhances robustness in multimodal reasoning tasks [44], [82], [91]. Furthermore, compositional methods can incorporate external knowledge or common sense when needed, but do so in a controlled and interpretable manner, avoiding the conflation of visual and linguistic content.

3.6 Lower Data Requirements and Improved Efficiency↩︎

Traditional monolithic models often require massive datasets and computational resources to achieve acceptable performance, particularly as training must account for the combinatorial explosion of possible visual-linguistic combinations [29], [30]. This approach is not only costly but also inefficient, especially in dynamic or specialized domains. In contrast, compositional visual reasoning promotes modularity and reusability: once basic visual skills (, object detection, OCR, segmentation) are learned, they can be reused across tasks without retraining [92]. This modular structure reduces the demand for extensive annotated datasets and enables few-shot or zero-shot generalization to new tasks by recombining known components [1], [70], [93]. Additionally, techniques such as visual program distillation or intrinsic tool orchestration (, PaLI [94], Griffon-R [95]) allow compositional systems to approximate multi-step reasoning with reduced runtime and compute overhead in deployment [94], [95]. Together, these advantages position compositional visual reasoning as a promising paradigm for building more capable, trustworthy, and efficient multimodal systems that align more closely with human reasoning.

4 Key Stages of Compositional VR Paradigms↩︎

7.2pt

: explicit support; : partial or optional support; : generally absent.

In this section, we trace the evolution of compositional visual reasoning models in recent years, highlighting how each stage has progressively enhanced their ability to decompose complex queries, reason over structured visual elements, and generalize across novel compositions. These transitions, illustrated in Figure 2, capture the key methodological shifts that have shaped modern compositional visual reasoning systems.

Early monolithic large vision-language models like  [10], [12], [13], [62], [63], [96]–[102] adopt end-to-end architectures that directly map visual and textual inputs to answers. These models aim to learn human-like visual reasoning capabilities directly from data and perform well on perceptual and shallow reasoning tasks. However, they struggle with more complex visual reasoning challenges, such as spatial understanding and fine-grained grounding [103]–[105]. These limitations have driven the development of compositional visual reasoning approaches, which decompose the reasoning process into structured, interpretable steps. Figure 2 and Table 2 illustrate the key architectural transition from monolithic to compositional paradigms since \(2023\), and Figure 3 highlights representative models at each stage of this evolution. The recent surge of interest in compositional visual reasoning is largely rooted in the growing recognition of LLMs’ powerful reasoning capabilities in domains such as natural language, mathematics, and code [106]–[108]. These advances have inspired researchers to explore whether such symbolic and structured reasoning abilities can be extended to vision-language settings.

Recognizing monolithic approaches’ limitations, researchers began exploring whether the structured reasoning strengths of LLMs could be leveraged for visual domains. Given LLMs’ ability to decompose and solve complex problems in language, math, and code, a natural question arose:

Stage I: Prompt-Enhanced Language-Centric Methods provides an effective solution to Question I. These approaches leverage LLMs’ strong language-based reasoning ability to break down complex visual questions into a series of simpler sub-questions, each of which can be independently answered by a VLM. After obtaining the sub-answers, the LLM performs final reasoning and synthesis entirely in the language space to produce the overall answer. This idea has given rise to compositional visual reasoning as a prominent research direction. Rather than treating reasoning as a single-step mapping from input to output, compositional visual reasoning paradigms frame it as a multi-stage process involving perception, task decomposition, intermediate reasoning, and final synthesis, as illustrated in Figure 2.

Although Stage I prompt-enhanced language-centric methods have achieved some success, they remain limited in visual understanding due to weak grounding in the underlying visual content and their relatively fixed prompting logic. This naturally raises an important question:

Stage II: Tool-Enhanced LLMs have emerged as a new paradigm to solve Question II. In this paradigm, the LLM issues grounding actions to external tools, and an external environment executes these actions to extract relevant information from the visual input. This interaction enables the system to progressively gather visual evidence and address complex visual reasoning tasks more effectively.

These methods offer greater reasoning flexibility, but often struggle with effective tool coordination, frequently resulting in noisy or suboptimal workflows. In most cases, the tool action planner, usually an LLM, has access to visual information only in the form of text descriptions produced by upstream vision models. If these textual descriptions are incomplete or inaccurate, the reasoning quality degrades and cannot be fully recovered downstream. This raises a natural question:

Stage III: Tool-Enhanced VLMs have emerged consequently as an answer to Question III, allowing the planner to access visual input directly rather than relying solely on textual descriptions. This direct perception reduces semantic loss, enables more accurate and informed action planning, and opens new possibilities for designing both the format of grounding actions and the structure of visual feedback. In some cases, this enables models to simulate visual imagination and perform visual verification by generating or manipulating visual content via tools during the reasoning process.

While prompt-enhanced and tool-enhanced methods rely on explicit problem decomposition and inter-module interaction to support compositional reasoning, this motivates the following question:

Stage IV: Chain-of-Thought VLMs are a promising solution to the challenges posed in Question IV, adopting unified architectures that tightly integrate perception and inference. These models perform multi-step reasoning without relying on external tools, and explicitly reveal intermediate thought processes and perception information before the final answer.

While chain-of-thought (CoT) VLMs integrate perception and reasoning, they remain constrained by relatively rigid pipelines, single-pass forward architectures, and reliance on textual representations of thought. Rather than directly interacting with visual content (, observing specific image regions), these models typically convert visual information into text and reason over it, limiting their ability to dynamically perceive and reflect. Based on these trends, a critical research question arises:

This question lies at the heart of Stage V: Unified Agentic VLMs, which represent the latest evolution in the field. These models incorporate higher-order cognitive mechanisms such as planning, memory, visual operation, imagination, textual feedback and visual evidence. Such capabilities enable iterative perception, step-by-step reasoning, and adaptive decision-making for complex visual tasks.

4.1 Stage I: Prompt-Enhanced Language-Centric Methods↩︎

6.5pt

Task Struct. denotes explicit question decomposition, stage-wise prompting, or structured evidence organization. : explicit support; : partial or implicit support; : generally absent.

We introduce prompt-enhanced language-centric approaches in this subsection, with an overview illustrated in Figure 4 and Table 3. These methods capitalize on the reasoning capabilities of LLMs, which typically serve two complementary roles. First, as a task decomposition module, the LLM decomposes a complex visual question \(q\) into a sequence of simpler sub-questions that can be independently addressed by VLMs given visual input \(v\). Second, as a consolidator or reasoner, it synthesizes a final answer based on the perceptual outputs or intermediate answers provided by the VLMs.

In this paradigm, the LLM performs the core reasoning process entirely within the language space, using information obtained from the VLM and textual query. The VLM is tasked with visual perception, either by directly answering decomposed sub-questions or by extracting relevant visual features from the input image to text descriptions. This modular design promotes interpretability, adaptability, and prompt-based control, and often functions effectively without requiring fine-tuning or task-specific supervision.

4.1.1 Task Decomposition Followed by Visual Perception↩︎

The typical architecture follows a structured pipeline in which a high-level visual question is first decomposed into a sequence of low-level sub-questions. Each sub-question is then independently addressed, and the final answer is synthesized based on the intermediate responses. Early methods like DDCoT [109], ChatCaptioner [113], IdealGPT [111] and Modeling Collaborator [114] adopt a straightforward strategy in which a frozen LLM decomposes a complex visual question into sub-questions or instructions, which are then processed by pretrained VLMs such as BLIP-2 [12]. These approaches are lightweight and task-agnostic, enabling rapid prototyping and broad applicability. However, their reliance on fixed prompts and lack of coordination training often results in suboptimal reasoning paths, redundant queries, and brittle performance.

4.1.2 Perceptual Grounding Prior to Reasoning↩︎

To address the limitations mentioned in Section 4.1.1, more recent work introduces varying degrees of structured interaction. CoLa [110] improves coordination by training the LLM to aggregate and reason over outputs from multiple VLMs, yielding better coherence and adaptability. PromptCap [112], MM-CoT [116], AdGPT [115], and Finedefics [117] shift the focus toward perception-first reasoning, where visual content is first converted into question-aware captions, dialogue-driven descriptions, visual rationales, or fine-grained evidence before LLM-based answer synthesis.

VCTP [118] strengthens this line by training the LLM to operate in a structured see-think-confirm pipeline. It adaptively grounds visual concepts, reasons over intermediate representations, and verifies outputs in a closed loop, improving transparency, consistency, and efficiency. CLiViS [122] extends perception-first reasoning to embodied visual reasoning, where an LLM planner coordinates VLM-based perception to build a cognitive map and evidence memory for final answer synthesis.

Recent KB-VQA methods further instantiate this perception-first paradigm by separating visual observation, knowledge retrieval, and language-based reasoning. QACap [119], NoteMR [120], and ReAG [121] respectively leverage question-aware observations, visual/knowledge notes, and reasoning-augmented retrieval to provide more reliable evidence for downstream LLM or MLLM reasoning. This separation enables compatibility with black-box LLMs and works well for knowledge-oriented VQA, but reasoning quality depends heavily on caption fidelity and lacks adaptive feedback.

4.1.3 Synthesis and Outlook↩︎

Overall, prompt-enhanced language-centric methods represent a clear progression from simple prompting strategies toward more coordinated and structured reasoning. However, these approaches remain limited in their visual grounding, as they rely solely on VLMs for perception without deeper integration of visual context. Additionally, their relatively fixed pipeline architecture constrains flexibility, making it difficult to adapt to more complex or dynamic visual reasoning tasks.

4.2 Stage II: Tool-Enhanced Large Language Models↩︎

In contrast to prompt-enhanced approaches that rely solely on textual reasoning, tool-enhanced LLMs equip language models with the ability to invoke external modules for perception, analysis, and computation. This paradigm typically involves two key components: generating actions from the current state, and transitioning between states by executing those actions.

The LLM acts as a central planner or action generator, responsible for decomposing tasks, selecting appropriate tools, and synthesizing final answers [29], [111], [123]. Specifically, it maps the current state to grounding actions by producing structured tool calls, such as code snippets or formatted prompts. The environment then interprets and executes these actions, triggering state transitions on the visual input using corresponding tools. These tools may include vision models (, detectors, captioners), knowledge bases, image editors, or code interpreters. Therefore, LLM’s role is to understand tool descriptions, generate calls to those tools, and interpret returned results in a multi-turn or sequential manner.

4.2.1 Single-Turn Framework↩︎

Representative systems such as GPT4Tools [124], Visual ChatGPT [125], Chameleon [69], VisProg [67], ViperGPT [126], HuggingGPT [127], CRAFT [128], Contextual Coder [129] and InternGPT [130] follow this structure, as illustrated in the tool-enhanced LLM framework overview in Figure 5 and Table 4. They assume access to a library of vision-language tools and use a frozen LLM to plan tool usage via prompt engineering, code generation, or natural language APIs. This setup enables training-free or zero-shot compositional reasoning across modalities, and provides a flexible interface to integrate new tools. However, these frozen LLMs often lack true tool awareness. They struggle to reason about tool capabilities, generate incorrect tool calls, and are unable to adapt when tool outputs are noisy or invalid [131].

4.2.2 Adaptive Tool Use via Training, Feedback, and Verification↩︎

While early tool-enhanced LLMs mainly rely on one-pass tool invocation, later methods shift toward adaptive tool use by introducing iterative feedback, tool-aware training, and structured executable reasoning.

One line of work closes the loop between tool execution and LLM reasoning. MM-REACT [132] serializes vision-expert outputs as textual observations and feeds them back to ChatGPT, enabling iterative action-observation reasoning without updating the LLM itself. Building on this feedback-driven view, CLOVA [133] and HYDRA [29] further turn tool use into an adaptive agentic process: CLOVA couples tool-augmented inference with reflection and continual tool updates, whereas HYDRA combines an LLM planner, a reinforcement-learning controller, and a reasoning engine to dynamically refine tool usage through interaction.

Another line improves tool awareness through training rather than prompting alone. VisRep [83] and DWIM [71] both use feedback from tool execution to teach LLMs how to reason with external modules. VisRep performs self-training, implemented as supervised fine-tuning, from program execution feedback, while DWIM uses discrepancy-aware workflow generation and instruct-masking fine-tuning to help the model understand tool functionalities, detect potential errors, and revise decisions.

A complementary line constrains tool use through structured executable reasoning. LEFT [36], NAVER [134], SYNAPSE [135], ViUniT [136], VADAR [137], TANGO [138], and PMA in CityEQA [139] all move beyond unconstrained tool invocation by imposing executable structures over the reasoning process. LEFT reasons over LLM-proposed visual concepts with a differentiable logic executor, while NAVER and SYNAPSE integrate LLM-guided logic with structured tool invocation and verification. ViUniT validates visual programs using visual unit tests. VADAR dynamically constructs Pythonic APIs and uses a test agent to validate generated functions; TANGO composes executable programs from navigation and exploration primitives; and PMA adopts a hierarchical Planner–Manager–Actor architecture with an object-centric cognitive map for long-horizon visual reasoning.

4.2.3 Synthesis and Outlook↩︎

Overall, this class of methods reflects a shift from static prompting to interactive, interpretable, and dynamically evolving tool use, where LLMs are not just passive planners, but increasingly learn to reason with and through tools [71]. These advances lay the groundwork for future multimodal agents with stronger compositionality, reliability, and generalization.

However, these methods have a critical bottleneck in that there is an unbounded number of interpretations from a finite set of visual building blocks. When LLMs operate solely on textual descriptions converted from visual inputs, their abstraction capacity becomes a major limitation. If the abstraction is inaccurate, misaligned, or fails to capture essential visual cues, crucial information may be lost during the conversion. This impairs the system’s ability to solve the task correctly, as important visual semantics are never incorporated into the reasoning process.

3.0pt

Planner denotes the central module that selects, invokes, or coordinates tools. Tool Interface specifies whether tools are invoked through direct tool calls, executable programs, multi-step workflows, or latent routing. Control distinguishes single-pass execution from iterative replanning using intermediate observations. State / Feedback summarizes whether the agent maintains memory (Mem.), receives visual feedback (Vis.), uses textual observations (Text obs.), or performs verification (Ver.). PnP Scope distinguishes training-free wrappers (TF wrapper), trained tool users, trained plug-in reasoners, trained tool-use policies, latent-coupled architectures, and multi-agent plug-ins. Core Training refers only to the planner, reasoner, or tool-use policy, excluding training of external tools or low-level modules.

4.3 Stage III: Tool-Enhanced Vision-Language Models↩︎

Tool-enhanced VLMs extend tool-enhanced LLMs by replacing the underlying LLM with a VLM, enabling more direct and flexible visual interaction, as illustrated in Figure 5 and Table 4. Unlike tool-enhanced LLMs where planners rely on preprocessed visual information from tools, this framework allows planners to directly receive raw images as input, reducing information loss and improving efficiency. Large-scale VLMs are dynamically augmented with external tools, enabling selective invocation of specialized systems (, object detectors, OCR engines, segmenters, and image generators) via natural language interfaces or latent control signals. This architecture enhances performance on complex tasks and mitigates issues such as hallucination and weak visual grounding.

Tool-enhanced VLMs can be broadly classified based on the modality and explicitness of their tool interaction. The first category, referred to as language-mediated control, includes models that issue interpretable instructions, such as natural language prompts or symbolic commands, to activate tool functionalities. The second category, termed embedding-mediated integration, encompasses models that coordinate tool use through learned latent representations, embedding control logic directly into the architecture without relying on explicit symbolic mediation.

4.3.1 Language-Mediated Grounding Action Control↩︎

Models such as Image-of-Thought [86], P2G [79], LLaVA-Plus [140], Syn [141], VIREO [84], and GCA [147] exemplify VLM-centered planning paradigms. These systems rely on the VLM to generate interpretable grounding actions, including natural-language prompts, symbolic commands, code-like programs, or formal constraints, which are then used to invoke relevant external tools for perception and reasoning.

For example, Image-of-Thought [86] simulates human cognitive processes by planning sequences of visual operations like segmentation or cropping and combining the resulting visual rationales with textual reasoning. P2G [79] facilitates fine-grained visual understanding by allowing the VLM to dynamically request help from OCR or grounding agents, particularly for high-resolution and text-rich images. LLaVA-Plus [140] employs a modular design in which the VLM acts as a planner that generates “Thought-Action-Value” prompts to control a repository of pre-trained visual tools. VIREO [84] and LATTE [142] improve multi-step reasoning by automatically decomposing questions and invoking tools step-by-step to resolve sub-questions.

Furthermore, with the emergence of GPT-o1 and DeepSeek-R1, reinforcement learning-based approaches for tool selection and intermediate evidence filtering have gained traction. OpenThinkIMG [150] and VToolR1 [146] optimize visual tool use for image-based reasoning, while Wiki-PRF [145] extends this idea to retrieval-augmented visual reasoning by training a VLM to dynamically invoke visual tools, retrieve multimodal knowledge, and filter irrelevant evidence using reward signals.

4.3.2 Embedding-Mediated Grounding Action Control↩︎

By contrast, VITRON [143] and VisionLLM v2 [144] exemplify models in the second category, where interaction with external modules occurs through learned embeddings rather than textual prompts. These systems embed communication pathways directly into their architecture. VITRON [143] employs a hybrid message-passing framework that combines discrete routing tokens with continuous embeddings to activate various back-end modules for tasks like image editing, segmentation, and video generation. VisionLLM v2 [144] introduces a “super link” mechanism that routes task-specific information using learnable query embeddings bound to routing tokens, allowing seamless integration of decoders for diverse tasks such as object detection, pose estimation, and image synthesis.

4.3.3 Visual Feedback in Tool-Enhanced VLMs↩︎

A more general class of tool-enhanced approaches for compositional visual reasoning aims to incorporate feedback during multi-turn, iterative processes. The most common form of feedback is textual, often delivered through natural language messages or structured programming outputs. However, when VLMs are used for action grounding and receive raw image inputs directly, this enables the system to obtain visual feedback in the form of updated or modified images. This gives rise to a distinct framework in which the model interacts with external tools to operate on images and then processes the resulting visual input for further reasoning.

A related direction introduces agent-wise collaboration and self-correction into VLM-centered tool use. MACT [151] decomposes visual reasoning into planning, execution, judgment, and answer agents, where the judgment agent verifies intermediate results and redirects earlier agents for revision, enabling adaptive test-time scaling and stronger factual grounding.

Models like Visual Sketchpad [148], VToolR1 [146], Self-Imagine [152], Creative Agents [149], and GCA [147] extend the visual-feedback paradigm by using external tools to synthesize imagined, modified, or intermediate visual states during reasoning.

4.3.4 Synthesis and Outlook↩︎

While tool-enhanced VLMs demonstrate strong generalization, grounded reasoning, and modular interpretability, they also present distinct challenges. These include high computational demands, increased input complexity, reliance on external tool accuracy, and architectural overhead [62], [79], [124], [144]. Future work should focus on improving integration robustness and efficiency—moving from prompt-based orchestration toward seamless architectural unification—to enable scalable and general-purpose multimodal agents [153].

4.4 Stage IV: Chain-of-Thought Reasoning VLMs↩︎

2.5pt

Subtype denotes the main mechanism used to elicit or learn CoT reasoning: Prompt-CoT, RL-CoT, visually grounded CoT (VG-CoT), geometry-distilled CoT (Geo-CoT), cross-view CoT (XView-CoT), and unified end-to-end CoT (Unified-CoT). Trace Form summarizes the intermediate reasoning representation, such as textual stages, scene graphs, visual programs, grounded tuples, segmentation chains, geometry traces, 3D latent traces, cognitive maps, or unified understanding-thinking-answering traces. Evidence uses controlled labels: Img = image-level evidence; ObjRel = object-relation structure; Box = bounding boxes or region references; Mask = segmentation masks; Token = learned perception tokens; ToolTrace = tool-generated traces; VisTrace = explicit visual evidence or grounded observations; Geo3D = geometric or 3D priors; MultiView = sparse or cross-view observations; HiRes = high-resolution visual evidence. Core Signal denotes the dominant learning or supervision signal: Prompt, SFT, Distill, RL, or SFT+RL.

Recent developments in vision-language models (VLMs) have led to the emergence of a new generation of chain-of-thought (CoT) reasoning-centric, end-to-end architectures as illustrated in Table 5 and Figure 6. Unlike prompt-enhanced or tool-enhanced systems that rely on external modules and prompt chaining, these models aim to integrate perception and reasoning into a single forward pass, leveraging pretraining and fine-tuning strategies inspired by large-scale LLM developments, such as GPT-o1 [170] and DeepSeek-R1 [171]. This section categorizes these models into four evolving trends: 1) Prompt-enhanced CoT reasoning VLMs; 2) RL-enhanced CoT reasoning VLMs; 3) Visually grounded CoT reasoning VLMs; 4) Geometry- and Cross-view-grounded CoT Reasoning VLMs.

4.4.1 Prompt-Enhanced CoT Reasoning VLMs↩︎

Prompt-enhanced CoT reasoning VLMs simulate multi-step reasoning behavior within a standard vision-language pipeline by leveraging in-context prompting or explicit stage-wise instruction formats. These models often frame the problem through intermediate representations or structured sub-tasks that guide reasoning while avoiding external modules or extra learning. A representative example is LLaVA-CoT [154], which breaks its output into structured reasoning stages, including summarization, captioning, intermediate deduction, and final conclusion. Similarly, CCoT [155] enhances compositionality by instructing the model to generate scene graphs from the input, which then guide the reasoning process through object-centric abstractions. GPT-o1-style prompting strategies extend this approach by defining explicit intermediate stages that structure the reasoning process in a predetermined manner. While these methods demonstrate improved performance and explainability through fixed structures, their reasoning capacity remains constrained by limited adaptability and generalization across diverse tasks.

4.4.2 RL-Enhanced CoT Reasoning VLMs↩︎

Inspired by proprietary reasoning models such as GPT-o1 [170] and DeepSeek-R1 [171], RL-enhanced CoT reasoning VLMs internalize systematic reasoning through combined supervised fine-tuning and reinforcement learning. These models typically adopt a two-phase learning framework: an initial supervised fine-tuning stage to encode reasoning patterns, followed by reinforcement learning or curriculum-based refinement to improve generalization and robustness. For example, Reason-RFT [160], Vision-R1 [162], VLM-R1 [161] and VisRL [159] leverage Group Relative Policy Optimization (GRPO) [172] to balance logical coherence with perceptual grounding during reasoning. KARL [156] introduces a cognition-perception synergy where reasoning accuracy and visual comprehension mutually enhance each other. G1 [157] adapts these strategies for visually interactive settings, enabling perception-grounded decision-making across complex tasks. Ground-R1 [158] introduces interpretable reasoning by decoupling visual evidence grounding from answer generation, trained end-to-end using only question-answer pairs without external annotations.

The two-phase learning framework aligns with findings that RL can encourage VLMs to actively search for correct answers [158], [171]. From an optimization standpoint, the performance improvements from RL (, RLHF) stem from how it reshapes the model’s output distribution. In distributional terms, the SFT objective corresponds to trajectory-level distribution matching using forward KL divergence (\(KL(P_{data} \Vert Q_{\theta})\)), encouraging the model \(\theta\) to broadly match the empirical distribution of the training data. While maximizing this distributional coverage ensures diversity, it also assigns probability mass to plausible but suboptimal or irrelevant outputs. In contrast, RL shifts the objective toward mode-seeking via reverse KL divergence (\(KL(Q_{\theta} \Vert P_{data})\)), training the model to concentrate specifically on generating outputs that are consistently correct or optimal. Although this reduces behavioral diversity, it yields more coherent and accurate responses, which is particularly beneficial in multi-step reasoning tasks where correctness and logical consistency outweigh broad coverage [173]–[175].

Although these models represent a significant shift toward more autonomous and cognitively aligned reasoning architectures, they often require substantial computational resources and careful curriculum design to achieve robust performance. While both RL-enhanced and prompt-enhanced CoT reasoning VLMs provide interpretable solutions by exposing structured reasoning steps, they still depend on implicit reasoning paths and typically lack explicit intermediate supervision or visual grounding.

4.4.3 Visually Grounded CoT Reasoning VLMs↩︎

Visually grounded CoT reasoning VLMs explicitly couple reasoning steps with visual grounding information, producing interpretable intermediate outputs such as bounding boxes, object references, or spatial annotations alongside final answers. This class of models benefits from training data that aligns structured reasoning with grounded visual supervision, either synthesized via tool-based pipelines or adapted from existing multimodal datasets  [68], [176], [177].

A representative example is PaLI-X-VPD [94], which introduces visual program distillation to inject coherent, tool-generated chain-of-thought reasoning traces into the training process. Building on the idea of explicit object grounding, VoCoT [82], LLaVA-Aurora [163] formulates each reasoning step as a tuple combining a visual region, its spatial coordinates, and a semantic description, enabling object-centric, multi-step inference. MM-GCoT [88] and VISTAR [164] contribute a dedicated dataset designed to enforce answer-grounding consistency, encouraging VLMs to align each reasoning step with grounded visual evidence. CoReS [165] advances this line by combining segmentation and logical inference through interleaved dual chains, improving compositional and fine-grained visual reasoning.

Despite their interpretability and stronger grounding, these models often depend on costly training pipelines and high-quality grounded traces, which may limit scalability and generalization in less structured environments.

4.4.4 Geometry- and Cross-view-grounded CoT Reasoning VLMs↩︎

Geometry-distilled and cross-view CoT reasoning VLMs extend Stage IV by injecting spatially structured evidence into the internal reasoning process of VLMs. Unlike visually grounded CoT methods that mainly supervise intermediate steps with explicit 2D evidence such as boxes, masks, or object references, this direction distills or constructs geometric priors from depth cues, 3D representations, viewpoint transformations, and limited-view observations. As a result, reasoning is guided not only by visible regions or textual rationales, but also by implicit spatial structures that encode depth, layout, occlusion, viewpoint consistency, and cross-view correspondences.

GeoThinker [166] actively integrates geometric evidence for spatial inference, while 3DThinker [167] grounds reasoning in 3D geometric imagination from limited views. MindCube [169] and VIEW2SPACE [168] further extend this paradigm to cross-view spatial reasoning, where grounded traces may take the form of cognitive maps, viewpoint-aware evidence, or multi-view spatial correspondences. Together, these methods move CoT VLMs from language- or region-grounded reasoning toward geometry-aware spatial reasoning, where 3D priors and cross-view evidence serve as implicit grounding signals for compositional inference. However, their reliance on geometric supervision, 3D foundation models, simulation data, or limited-view annotations may constrain scalability, and their latent spatial reasoning process is often less interpretable than explicit grounding traces.

4.4.5 Synthesis and Outlook↩︎

CoT reasoning VLMs bridge perception and inference by exposing intermediate reasoning steps. Prompt-enhanced and RL-enhanced methods improve the structure and reliability of reasoning, while visually grounded CoT models align intermediate steps with explicit 2D evidence such as boxes, masks, or object references. More recent geometry-distilled approaches further push this paradigm by injecting 3D grounding signals into VLMs, encouraging models to produce CoT traces that encode depth, layout, viewpoint consistency, and latent spatial structure. The next frontier is therefore to develop CoT VLMs whose reasoning is not only linguistic and visually grounded, but also geometrically aware. However, the predominantly single-pass nature of current CoT architectures still limits iterative evidence inspection, self-correction, and refinement for complex visual reasoning tasks.

4.5 Stage V: Unified Agentic Vision-Language Models↩︎

3.0pt

Unified agentic VLMs represent a recent shift in multimodal reasoning, wherein the model actively plans, adapts, imagines, and executes a sequence of decisions to solve complex visual tasks [178], [181]. Different from prompt-enhanced or tool-augmented systems, where language models issue static queries or deterministic tool calls, unified agentic VLMs are designed to autonomously decompose tasks, selectively gather visual evidence, and iteratively refine their reasoning process. These models exhibit a form of learned deliberation: they can identify uncertainty, determine what additional visual evidence is needed, and justify their answers through explicit reasoning and action trajectories, as illustrated in Figure 7 and Table 6.

These models [105], [179] often operate within a multi-step feedback loop that integrates perception, decision-making, execution, and memory. A typical architecture consists of a controller, usually an LLM, VLM, or learned policy model, together with visual engines or internal visual operations, such as captioning, segmentation, detection, grounding, zooming, OCR, or region retrieval, and an intermediate memory or context accumulation module. The reasoning process can be conducted explicitly, through structured chains of visual operations, or implicitly, through learned state representations. Accordingly, Stage V methods can be broadly organized into two directions. The first emphasizes explicit visual exploration, where models discover informative regions, zoom into image patches, perform visual operations, and accumulate evidence in memory to better exploit visual information already present in the input. The second emphasizes visual-state-augmented reasoning, where models maintain latent visual states, imagined representations, or world-model-like dynamics to support internal imagination, belief refinement, and reasoning over information that is not directly observed.

4.5.1 Automatic Discovery of Informative Regions and Goal-Driven Exploration↩︎

A subset of unified agentic VLMs demonstrates the ability to automatically identify, search, and discover informative regions within an image to iteratively solve subtasks and ultimately complete the overall task. These models exemplify active visual reasoning by dynamically allocating attention and modifying inputs based on task demands.

Representative systems illustrate this paradigm in diverse ways. SEAL [178] incorporates a visual working memory that stores the question, global image, target regions, and retrieved patches. When the initial image context proves insufficient, the model identifies missing visual entities, performs targeted search and cropping, and revisits the question with enriched inputs. ZoomEye [180] treats the image as a hierarchical patch tree and uses confidence scores from an MLLM to guide zoom-in actions and determine when to stop based on answer certainty. FAST [181] adopts a dual-system architecture that switches between fast and slow reasoning modes. When greater precision is needed, it activates built-in visual operations, such as segmentation and proposal generation, and aggregates the resulting evidence through a chain-of-evidence structure.

Other methods, such as CogCoM [105], VisCoT [184], Insight-V [183], GeReA [182], ViGoRL [185], SIFThinker [186], Argus [27] and DeepEyes [187] guide an LLM to generate a sequence of visual manipulation steps—such as grounding, zooming, OCR, and counting—to incrementally acquire task-relevant information. DC2 [179] proposes an alternative strategy by dividing high-resolution images into patches, summarizing them individually, and using a visual memory mechanism to selectively retrieve relevant regions at inference time.

4.5.2 Latent Imagination and World-Model-Augmented Reasoning↩︎

Another class of unified agentic VLMs augments reasoning with latent or imagined visual states rather than only inspecting visual evidence already present in the input. These methods use latent visual tokens, hidden visual representations, world-model dynamics, or simulator-generated observations to support internal imagination and reasoning beyond directly observed visual content.

Some models focus on textual imagination within the latent space. For example, Mirage [188] introduces a machine mental imagery framework that augments VLM decoding with latent visual tokens alongside textual outputs. Rather than generating pixel-level images, the model performs “visual thinking” by extending hidden state trajectories with multimodal tokens, maintaining a multimodal reasoning flow entirely in latent space. Another notable approach, FOREWARN [189], adapts the LLaMA-3.2-11B-Vision-Instruct model by replacing its image tokenization module with a world model encoder and latent dynamics component. This design allows the model to internally simulate visual states during reasoning, enabling it to perform latent imagination without relying on external rendering. Moving from latent simulation to explicit simulator-mediated imagination, Astra [190] actively requests novel-view observations from an integrated world simulator. It uses a world simulator to produce novel-view visual observations, allowing the model to reason over unobserved spatial layouts and cross-view evidence beyond the original input.

4.5.3 Synthesis and Outlook↩︎

Unified agentic VLMs enhance compositionality, adapt to ambiguous inputs, and support interpretability through intermediate visual states, visual memories, or visual artifacts [105], [179], [181], [187]. Many show strong performance on high-resolution and compositional benchmarks such as V*Bench [178], GQA [68], ReasonSeg [191], and MM-Vet [192]. They also demonstrate robustness when paired with relatively small LLMs, offering an efficient alternative to large proprietary models [95].

Nonetheless, these systems face challenges. Many rely on handcrafted prompts, special tokens, or rule-based triggers, which limit scalability [140], [165]. Multi-step pipelines often incur higher computational costs and may propagate errors through sequential stages. Visual operations such as cropping or segmentation are sometimes imprecise, especially for small or occluded objects [71], [133], [180]. Moreover, the lack of unified supervision across reasoning steps and final outputs hinders end-to-end optimization [94], [158].

In summary, agentic vision-language models represent a promising direction that combines planning, perception, and reasoning within an integrated framework. They bridge the gap between passive captioning and interactive visual understanding. Future work may focus on improving their decision policies, unifying learning objectives, and integrating them with broader multimodal toolchains to support general-purpose visual intelligence.

5 Benchmark and Evaluation↩︎

Evaluation and benchmarking play a pivotal role in assessing the progress and diagnosing the limitations of visual reasoning systems. A wide range of datasets and metrics have been developed to probe different aspects of reasoning, minimize dataset bias, and analyze model capabilities with fine-grained annotations.

5.1 Types of Benchmarks and Datasets↩︎

This subsection presents examples of different types of benchmarks and datasets, as illustrated in Figure 8.

General Visual Question Answering. General VQA benchmarks aim to evaluate a model’s ability to answer natural-language questions based on real-world images. These datasets focus on broad coverage of visual concepts and diverse question types (, object presence, color, location, and actions), usually with human-generated questions. Prominent examples include VQA-v1 [193], VQA-v2 [194], COCO-QA [195], Visual Genome [196], and Visual7W [197]. These datasets focus broadly on object presence, attributes, relationships, and basic visual comprehension. Variants such as VQA-CP [198] specifically address biases through carefully restructured training and testing splits. In short, general VQA datasets are mostly well-suited for testing perception-based understanding grounded in image content alone.

Relational Scene Graph Reasoning and Systematic Generalization Relational scene graph reasoning and systematic generalization datasets are designed to evaluate a model’s ability to reason over novel combinations of familiar visual concepts, or interpret high-level semantic concepts across unseen environments. These tasks require models to perform multi-step inference, manipulate symbolic representations, and handle compositional semantics such as nested attributes, spatial relations, and logical structures. Often constructed using synthetic or structured environments, these benchmarks allow precise control over data generation and enable evaluation of reasoning fidelity and generalization capabilities under distribution shifts.

Systematic generalization is rigorously tested by datasets such as GQA [68], ReasonSeg [191], TallyQA [199], NaturalBench [89], Cola [200], the Visual Abstractions Benchmark [201], CREPE [64] and SugarCrepe [202]. These datasets explicitly target the models’ compositional generalization, relational learning, and reasoning under uncertainty. In short, this category is essential for assessing structured reasoning, generalization to novel combinations, and step-wise inference fidelity.

Synthetic and Diagnostic Evaluation. Synthetic and diagnostic benchmarks provide controlled environments to test specific reasoning sub-skills, such as attribute comparison, spatial relation tracking, and logical consistency. These datasets are often built with artificial scenes, minimizing irrelevant variability and allowing fine-grained analysis of reasoning behaviors. Diagnostic variants further introduce linguistic perturbations, attention supervision, or adversarial rephrasings to evaluate robustness and interpretability [37], [203].

Representative datasets include CLEVR [5], SHAPES [204], TaskMeAnything [205], SVRT [206], CLOSURE [207], CURI  [208] and CLEVR variants (, CLEVR-Ref+ [209], CLEVR-XAI [210], QLEVR [211], CLEVR-X [212], CLEVR-Dialog  [213]). Diagnostic benchmarks like VQA-X [214], VQA-HAT [215], and VQA-Rephrasings [216] specifically target reasoning quality, explanation generation, human attention alignment, and linguistic robustness. Additionally, synthetic tests for abstract reasoning also fall under this category, notably, Raven’s Progressive Matrices datasets (, PGM [217], RAVEN [7]). Other examples include KANDINSKYPatterns [218] and Bongard-LOGO [219], which focus on structural pattern recognition and analogy-based concept learning, respectively.

Knowledge-Based and Commonsense Reasoning. Knowledge-based and commonsense benchmarks are developed to assess a model’s capacity to incorporate external knowledge (, factual, cultural and commonsense) into the visual reasoning process. Unlike general VQA, these tasks require inference beyond observable pixels, leveraging external sources or prior training on world knowledge.

To this end, a number of datasets have been proposed to evaluate such capabilities. Examples include OK-VQA [176], A-OKVQA [177], VCR [66], FVQA [220], and KB-VQA [221], which require models to reason with factual or commonsense knowledge not present in the image. In addition, CRIC [222] further emphasizes compositional and grounded commonsense reasoning. These benchmarks are therefore crucial for evaluating whether a model can move beyond surface-level perception and engage in truly knowledge-intensive visual inference.

Image-Based Text and Document Understanding. These benchmarks focus on evaluating a model’s ability to extract, read, and reason over textual information embedded in visual formats such as photographs, charts, and scanned documents. Tasks typically involve OCR-based understanding, document layout parsing, or diagram interpretation, requiring alignment between textual and spatial structures. Representative datasets include TextVQA [223], ST-VQA [224], DocVQA [225], and AI2D [226].

High-Resolution and Fine-Grained Perception. To explore vision-language models’ capabilities in fine-grained perception, high-resolution benchmarks such as V*Bench [178], HR-Bench (4K/8K) [179], and LISA-Grounding [191] evaluate models’ performance on detailed visual scenes and precise object recognition.

Multidisciplinary and Specialized Domains. Benchmarks under this category extend visual reasoning into specialized domains or underrepresented user populations. They include medical diagnostics, accessibility for the visually impaired, cultural diversity, mathematical and chart reasoning, and embodied agents [227]–[233]. Many also incorporate multimodal challenges across domains.

Specialized datasets are developed for specific domains and applications, including medical imaging (, VQA-RAD [227], PathVQA [228]), visually impaired assistance (, VizWiz [229]), multilingual and cultural diversity (, WorldCuisines [230]), mathematical reasoning (, MathVista [234], ChartQA [231], Geoclidean [233]), and embodied environments (, VQA 360° [232]). Additionally, comprehensive benchmarks such as MM-Vet [192], MME [235], MMMU [236], and SEED-Bench [237] systematically evaluate vision-language capabilities across multiple reasoning domains. Benchmarks including MMBench [238], LLaVA-Bench [239], NovPhy [240], M\(^3\)CoT [241], and VisIT-Bench [242] further probe fine-grained skills such as instruction-following, physical reasoning, and visual conversational interactions.

Hallucination and Robustness Evaluation. This category includes benchmarks designed to detect hallucinated outputs, breakdowns in robustness, and model inconsistency, particularly in cases where the alignment between vision and language inputs is unreliable or weak. These tasks typically involve verifying whether model-generated responses are faithfully grounded in the input image and whether the model maintains consistent behavior under various perturbations. Representative datasets such as POPE [243], HallusionBench [244], YESBUT [245], and CausalVQA [246] are designed to ensure that model outputs remain factually supported by visual evidence.

5.2 Evaluation Metrics↩︎

Evaluating visual reasoning involves a diverse set of metrics designed to capture not only the correctness of answers but also the interpretability, grounding consistency, and efficiency of the reasoning process. These metrics are typically selected based on task characteristics, dataset design, and the specific reasoning abilities being assessed.

Accuracy-Based Evaluation. Accuracy and exact match remain the dominant metrics for classification-based tasks, including general VQA, compositional reasoning (, GQA [68]), and knowledge-intensive VQA (, OK-VQA [176], A-OKVQA [177]). For tasks with variable answers, such as TextVQA [223] and ST-VQA [224], metrics like VQA Score are used. A common evaluation paradigm in these benchmarks is option matching, a closed-set method where models select from predefined candidates. This reduces ambiguity in open-ended tasks and ensures standardized evaluation. Datasets such as CLEVR [5], ScienceQA [247], MME [235], and MMBench [238] rely on this approach to improve fairness and comparability.

Grounding and Segmentation Metrics. For tasks requiring visual localization, metrics such as Intersection over Union (IoU), and generalized IoU [248] are widely used to assess the alignment between predicted and ground-truth regions. Metrics like AP@50 and AP@75 evaluate precision at different IoU thresholds. Tasks involving visual reasoning with segmentation leverage grounding accuracy and salient object detection metrics, including S-measure and mean absolute error.

Rationale and Text Quality. To evaluate explanation quality, metrics such as BLEU [249], ROUGE [250], and CLIP sentence similarity [251] are used to assess the coherence and informativeness of generated rationales. F1, precision, and recall are employed for sequential tasks or when answers involve structured text. Metrics like answer-grounding consistency and average output token length further probe whether generated explanations align with visual evidence and maintain concise reasoning.

Task-Specific and Behavioral Metrics. Visual reasoning systems are further evaluated using specialized metrics such as search length (number of steps in visual search) [178], action counts (frequency of operations like zoom or segmentation) [86], hallucination rate (frequency of unsupported content) [252], and confidence improvement (sensitivity to question variations) [37]. Keypoint-aware evaluation and format adherence metrics are employed in manipulation and program-based reasoning settings.

Efficiency and Cost. Finally, metrics related to resource usage, such as runtime, inference latency, LLM token consumption, and financial cost, are reported to assess the computational efficiency and scalability of vision-language models [134].

These evaluation metrics collectively provide a comprehensive view of model performance, capturing both end-task accuracy and the underlying reasoning behavior, interpretability, and resource efficiency necessary for reliable multimodal intelligence.

5.3 Synthesis and Outlook↩︎

Current benchmarks and evaluation protocols provide a broad landscape for assessing visual reasoning, offering evaluations across final answer accuracy, visual grounding, textual explanation quality, and domain coverage. They support tasks ranging from general VQA to fine-grained perception, knowledge-based reasoning, and multimodal interactions, spanning diverse modalities, task types, and reasoning skills. Despite this breadth, several critical limitations remain.

An important limitation in current evaluation practices is that existing benchmarks rarely evaluate explicit intermediate reasoning steps, also referred to as step-level or thought-step outputs, which have become increasingly central in recent compositional visual reasoning methods. Traditional evaluation metrics like BLEU [249], ROUGE [250], and grounding scores provide only indirect assessments by evaluating final rationales or explanations. Consequently, they fail to directly measure the accuracy, coherence, or causal consistency of individual reasoning steps. For instance, a model might produce correct final answers but rely on incorrect intermediate deductions or superficial correlations.

Second, current benchmarks lack explicit quantification and analysis of reasoning difficulty levels, making it challenging to understand the distribution of question complexity. Questions involving a single object and its attributes are significantly easier compared to those involving multiple objects with intricate attribute references and complex inter-object relationships. For example, a question like “What color is the ball?” demands far simpler reasoning than “Which object next to the small blue cube is the largest red sphere?” Without explicitly defined difficulty scales, researchers cannot accurately evaluate a model’s capacity to handle varying complexity levels.

To address these issues, future benchmarks should: 1) incorporate detailed annotations and corresponding metrics to evaluate intermediate reasoning steps, with a particular focus on assessing step-level consistency, causal coherence, and grounding accuracy; 2) explicitly quantify and categorize reasoning difficulty levels, enabling systematic evaluation of model performance across a spectrum of question complexities; and 3) provide comprehensive diagnostic insights by clearly differentiating reasoning capabilities, thereby helping researchers identify specific limitations in models and informing targeted improvements. Advancing these directions will enable deeper insights into model behaviors and support the robust development of compositional visual reasoning frameworks.

6 Insights, Challenges and Directions↩︎

Compositional visual reasoning (CVR) has evolved quickly in recent years, with numerous papers emerging across different classes of models and evaluated on a wide range of benchmarks. In this section, we synthesize key insights from recent research, identifying trends and promising principles. We then investigate core challenges that continue to limit scalability, generalization, and robustness, as well as examine future directions aimed at overcoming these challenges to advance general-purpose CVR systems.

6.1 Insights↩︎

Recent studies in compositional visual reasoning indicate a notable shift from static, perception-based approaches toward structured, multi-step reasoning frameworks inspired by cognitive science. Models such as Chain-of-Manipulations [105], and Visual CoT Prompting [118] introduce modular reasoning procedures that enable intermediate decision making and enhance interpretability. Dual-system frameworks, exemplified by FAST [181], emulate both intuitive and deliberative cognitive processes, while reinforcement learning–based methods like Ground-R1 [158] and Vision-R1 [162] leverage feedback signals to promote self-corrective and generalizable reasoning without heavy reliance on supervised annotations.

This trend is further exemplified by the emergence of agentic VLMs that integrate control flow, memory, and visual grounding into a unified reasoning loop. Architectures such as SEAL [178] and ZoomEye [180] enable iterative, goal-directed visual exploration guided by questions, while Griffon-R [95] fuses perception and reasoning within a single forward pass. These models demonstrate the potential of embedding planning, action selection, and evidence integration within an agentic framework for compositional tasks.

To address data scarcity, researchers have turned to automated pipelines that synthesize supervision using LLMs and vision models. These pipelines generate CoT rationales, visual masks, and object-level annotations, providing rich multimodal supervision. Benchmarks such as MM-GCoT [88] and VoCoT [82] emphasize step-wise reasoning and grounded verification, contributing to more interpretable and robust evaluation protocols.

At the architectural level, ongoing efforts aim to unify static scene understanding with dynamic interaction and planning capabilities. This includes the development of tool-aware LLMs for adaptive tool invocation and coordination, as well as models with visual working memory mechanisms that manage long-horizon reasoning over fine-grained visual inputs. Models like DC2 [179] and SEAL [178] illustrate the effectiveness of such memory-based designs in maintaining contextual coherence while minimizing computational overhead.

6.2 Key Challenges and Future Directions↩︎

Despite the rapid progress in compositional visual reasoning, significant challenges remain in enhancing reasoning capability, improving data quality, integrating architectures, and refining evaluation methodology. Addressing these limitations is essential for developing scalable, trustworthy, and general-purpose compositional visual reasoning systems.

6.2.1 Limitations of LLM-Based Reasoning↩︎

A central limitation across all five stages of compositional visual reasoning systems discussed is their heavy reliance on LLMs as the core reasoning module. While LLMs excel at processing natural language and generating coherent text, they often lack the grounded, structured reasoning necessary for complex visual tasks. In particular, LLMs lack an internal world model, which limits their ability to simulate visual or physical dynamics, perform spatial transformations, or reason about hypothetical or counterfactual outcomes. These capabilities are essential for tasks such as object reconfiguration, spatial prediction, and planning tool usage [253].

For example, when tasked with predicting whether one object can fit into another or reasoning about the consequence of moving an obstacle, LLMs tend to rely on linguistic priors rather than perceptual evidence, often producing plausible but incorrect answers. This limitation is rooted in their autoregressive architecture, which is optimized for next-token prediction rather than iterative self-correction or causal inference [254].

From a cognitive standpoint, this behavior resembles System 1 reasoning—fast, heuristic, and shallow—rather than System 2 reasoning, which involves deliberate, multi-step analysis grounded in perceptual input and symbolic manipulation [255]–[257]. This gap is especially evident in visual reasoning benchmarks that require multi-hop spatial inference, counterfactual exploration, or tracking of visual state changes, where LLMs often fail to reason beyond surface-level correlations [258], [259].

Potential Directions. To address these challenges, one promising solution is to introduce an explicit internal world model into compositional visual reasoning systems. Such models would enable agents to simulate hypothetical visual scenarios, reason over spatial and temporal dynamics, and plan multi-step actions grounded in perception. This capability supports forward simulation, counterfactual inference, and long-horizon planning. It also aligns more closely with System 2 reasoning. Incorporating world models takes inspiration from model-based reinforcement learning and offers a pathway to bridging the gap between language-driven reasoning and perceptually grounded intelligence [158].

6.2.2 Hallucinations↩︎

Although LLMs are highly effective at language-based inference, their reasoning often lacks a strong connection to visual content, particularly when visual inputs are projected into language space without sufficient grounding. This misalignment can lead to hallucinated conclusions, superficial associations, and inaccurate spatial or relational reasoning [79], [86]. For instance, when presented with an image of a green banana, an LLM may incorrectly respond that the banana is yellow, reflecting linguistic bias rather than perceptual evidence.

Compositional approaches, while modular in structure, are not fully immune to hallucination and unfaithfulness. Models can produce plausible yet unsupported intermediate steps or final answers, especially when reasoning components operate on incomplete or poorly grounded visual information [71], [95], [105]. These issues may arise from semantic mismatches between visual inputs and symbolic reasoning, limited supervision during grounding, or an overdependence on prior knowledge encoded in the LLM [39], [41].

Furthermore, shortcut learning persists even in multi-step pipelines, where models exploit spurious correlations within decomposed tasks rather than reason from visual evidence. These issues limit generalization, decrease interpretability, and reduce confidence in system outputs, highlighting the need for improved alignment, step-level supervision, and stronger grounding mechanisms in compositional reasoning frameworks [88], [181].

Potential Directions. Given the ambiguity and context-dependence of real-world CVR tasks, incorporating interactive frameworks with human-in-the-loop supervision holds promise. Allowing users to guide, verify, or revise intermediate reasoning steps can enhance both system reliability and user trust. Such interaction also supports adaptive learning by providing feedback signals that reinforce faithful reasoning.

6.2.3 Bias Toward Deductive Reasoning↩︎

Most current compositional visual reasoning systems predominantly rely on deductive reasoning, where inference steps follow formal logic. However, this approach assumes the correctness of initial premises and can produce erroneous conclusions when the inputs are noisy or biased [26], [260]. For example, a system might conclude that an object is a chair because it has four legs and a backrest, even if the object is actually a sculpture, due to flawed visual grounding.

Potential Directions. To improve robustness and adaptability, alternative reasoning paradigms offer promising directions. Inductive reasoning enables models to generalize patterns from visual observations—such as learning affordances from multiple object instances [261], [262]. Abductive reasoning supports generating plausible explanations when visual input is ambiguous, for instance, inferring that a spilled drink on the floor suggests a tipped-over glass nearby [263], [264]. Analogical reasoning facilitates relational transfer, such as applying knowledge about assembling furniture from diagrams to configuring unfamiliar mechanical parts in a robotics task, where similar spatial relations and assembly logic are required [265], [266].

6.2.4 Data Limitations and Scalability Challenges↩︎

Progress in compositional visual reasoning is significantly hindered by data-related constraints. Existing datasets lack sufficient compositional diversity, multi-step supervision, and grounded visual programs, limiting the development of robust reasoning capabilities [267], [268]. High-quality annotations for step-by-step inference are costly and difficult to scale [79], [162], [269], while synthetic pipelines introduce noise due to unreliable perception tools [71], [133].

Supervised fine-tuning on curated CoT data can cause overfitting and cognitive rigidity, restricting generalization [71], [94], [160]. Automatically generated instruction data often lacks reasoning depth and diversity, and even advanced models like GPT-4V can yield unfaithful or inconsistent outputs [105], [181]. Benchmarks such as CLEVR [5] and VQA [193] still emphasize shallow tasks, overlooking multi-step reasoning [88], [270].

Potential Directions. Addressing data scarcity and scalability challenges requires the development of more efficient and grounded data construction strategies. One promising direction is to build hybrid data engines that combine synthetic scene generation with real-world imagery, augmented by perception-enhanced simulation environments [271], [272]. These platforms can generate richly annotated multi-step reasoning traces with minimal manual effort. Additionally, leveraging weak supervision, self-training, and feedback-driven refinement can mitigate annotation costs while improving fidelity and reasoning diversity. Instruction tuning with human-in-the-loop filtering or discrepancy-aware self-verification can further enhance data quality [71], [83], [273].

6.2.5 Tool Integration and Architectural Bottlenecks↩︎

Compositional visual reasoning faces architectural challenges in the integration of LLMs with vision modules and external tools. Current systems often lack tool awareness, incur high computational costs during tool use, and struggle with re-planning based on intermediate results [29], [69], [71], [83]. Vision encoders introduce information bottlenecks by projecting low-resolution inputs into limited token spaces [60], [71], [178], while fragmented pipelines that treat static and dynamic content separately weaken cross-modal grounding [94], [133], [140], [143].

Tool learning is hindered by retrieval constraints, inconsistent tool formats, limited generalization to unseen tools, and underdeveloped multimodal capabilities [69], [274]. Balancing reasoning depth with efficiency remains an open issue [275].

Potential Directions. Future research should explore more integrated architectures that support tighter coupling between perception, reasoning, and tool execution. Promising directions include improving tool-awareness, designing efficient planning and re-planning strategies, enhancing cross-modal grounding, and supporting scalable generalization to unseen tools.

6.2.6 Benchmark Limitations and Evaluation Contamination↩︎

Current benchmarks often overestimate model capabilities by relying on in-distribution data and biased annotations [202], [276]. Models frequently exploit linguistic shortcuts instead of engaging in faithful visual reasoning [37], [273]. Evaluation contamination arises from synthetic queries, artificial distribution shifts, and browsing-enabled LLMs, which obscure reasoning failures [132], [252].

A particularly critical gap lies in the evaluation of compositional visual reasoning models. Although these methods emphasize explicit intermediate reasoning steps, current benchmarks typically assess performance based solely on final answers, without evaluating the quality or faithfulness of the intermediate reasoning process. This evaluation limitation undermines interpretability and hampers progress toward robust, generalizable visual reasoning [277].

Potential Directions. Future progress relies on the development of more nuanced evaluation protocols that move beyond final answer accuracy. Promising directions include difficulty-aware scoring, assessment of intermediate reasoning steps, and multimodal explanation evaluation. Moreover, extending benchmarks to include richer visual compositions, multi-hop reasoning trajectories, and step-level annotations will be crucial for advancing robust and generalizable compositional visual reasoning [45].

7 Conclusion↩︎

In this survey, we focus on recent advances in compositional visual reasoning (CVR), a rapidly emerging research direction that emphasizes structured, multi-step reasoning over visual inputs. Unlike monolithic vision-language models that directly map input to output in a single pass, CVR frameworks aim to explicitly bridge perception and reasoning through intermediate, interpretable steps. We highlight the key distinctions between these two paradigms, illustrating why compositional reasoning is essential for achieving robust and cognitively aligned intelligence.

This survey is organized around four core questions: Why is compositional visual reasoning necessary? What are the major architectural shifts and reasoning paradigms in this field? What benchmarks and evaluation protocols are currently used? What are the prevailing limitations and open challenges? To answer these questions, we reviewed over \(100\) papers and distilled the field into five major stages of development: prompt-enhanced language-centric methods, tool-enhanced LLMs, tool-enhanced VLMs, chain-of-thought VLMs, and unified agentic VLM-based frameworks. For each stage, we introduce representative works, analyze their design principles, and summarize their strengths and limitations. We also provide a roadmap that charts the evolution of CVR systems and offers insights for future research. In addition, we review benchmark datasets and metrics, and highlight key challenges such as visual grounding, hallucination, data scalability, system bottlenecks, benchmark limits, and evaluation contamination.

ACKNOWLEDGMENT↩︎

This work was supported by Building 4.0 CRC and the Commonwealth of Australia through the Cooperative Research Centres Program. It was also partially funded by the DARPA ANSR program (FA8750-23-2-1016) and the ARC DECRA program (DE250100032).

References↩︎