1 Introduction↩︎

External tools such as web search, OCR, table readers, PDF parsers, code execution, and visual operators expand the capability of vision-language models (VLMs) beyond in-context prompting [1]–[4]. In these settings, success depends not only on the final answer, but also on planning and executing grounded intermediate steps, including which tool to call, when to call it, and how to set its arguments from multimodal evidence.

One common approach is to train trajectory-tuned controllers on synthetic tasks and verified traces. MAT-AGENT [5] follows this approach by generating multimodal tool-use tasks, executing tools to collect successful trajectories, and applying supervised fine-tuning (SFT) to learn tool invocation. This strategy is effective, but it also ties the controller closely to supervised traces. When the policy deviates, for example under a new PDF layout or receipt style, the training signal offers limited guidance for recovery.

A complementary direction uses self-exploration. SPORT [6] alternates between sampling tool-use steps and verifying them, then converts rollouts into step-wise preferences without human labels. This introduces process-level feedback, but the supervision remains relatively unstructured. For example, SPORT does not explicitly encode multimodal chain-of-thought, grounding tags that link language to regions or cells, or verifier rationales. As a result, the learned preferences are often hard to interpret and can fail under distribution shift.

These approaches share two limitations. SFT-only tuning on expert trajectories tends to saturate quickly because the loss depends heavily on one teacher trace and provides weak signals for local repair. Standard reinforcement learning also provides limited recovery-oriented supervision in long-horizon tasks, because a scalar failure reward does not identify which decision should be revised.

We therefore train tool-using VLM agents with reflection-augmented reinforcement learning, where diagnostic reflections are optimized as explicit recovery signals. Instead of treating reflection as a test-time prompting heuristic, we model it as a learnable variable that links a failed action to its correction.

We introduce ReGRPO (Reflection-augmented Group Relative Policy Optimization) to realize this idea. On the data side, ReGRPO builds a Structured Reflective Data Engine that converts tool-execution failures into grounded Reflection-of-Thought (RoT) triplets. The engine perturbs MAT-AGENT trajectories [5], executes the perturbed actions to obtain real failure observations, and then generates structured reflections with ErrorType, Evidence, and FixPlan. On the optimization side, ReGRPO jointly optimizes reflection and correction tokens within the same local trajectory objective, which gives the policy step-level supervision beyond final success or failure. ReGRPO keeps the standard GRPO optimizer and adds structured reflection parameterization to enable end-to-end policy learning.

At inference time, ReGRPO follows the same principle that failures should trigger explicit local repair rather than blind continuation. We use a lightweight single-path, zero-verifier setup with a deterministic trigger, which opens a reflection-correction block only when needed. This design allows recovery from tool errors while keeping deployment efficient.

On GTA [7] and GAIA [8], under the same backbone and tool suite as MAT-AGENT and SPORT, ReGRPO consistently improves overall accuracy.

In summary, this paper makes three contributions:

• We build a Structured Reflective Data Engine that converts tool-execution failures into grounded Reflection-of-Thought supervision with ErrorType, Evidence, and FixPlan, paired with corrected actions.

• We present ReGRPO, which adopts the standard GRPO optimizer while adding structured reflection trajectory parameterization and a zero-verifier trigger, enabling joint optimization of reflection and correction tokens under reproducible single-path deployment.

• Under the same backbone and tool settings, ReGRPO achieves the strongest results among the compared open-source controllers on GTA and GAIA.

2 Related Work↩︎

2.1 Multi-Modal Agents↩︎

Multimodal agents [9]–[13] can solve complex problems via external tools and APIs. For example, CLOVA [14] uses LLMs as controllers to compose off-the-shelf visual tools. ViperGPT [1] uses code-generation models to compose vision-and-language models into subroutines to produce results for arbitrary queries. VideoAgent [2] adopts multi-step reasoning, where the agent selects tools according to intermediate observations.

Though LLM-driven methods can achieve strong results, VLM-driven agents [15]–[17] are often more efficient for visual tasks because the controller directly consumes images or videos and tool outputs. For example, GenArtist [17] proposes a unified image generation and editing system coordinated by a multimodal large language model (MLLM) agent.

In addition, other works [18]–[20] use models to generate AI feedback that improves performance. For example, LLaVA-Critic [18] presents a high-quality dataset tailored to follow instructions in complex evaluation settings, providing quantitative judgments and accompanying reasoning.

2.2 Datasets for Tool-Using Agents↩︎

The reasoning ability of VLM-driven agents is often weaker than that of large text-only LLMs. To bridge this gap, recent works synthesize tool-usage data to tune open-source VLMs [12], [21], [22]. For example, DEDER [23] uses in-context learning to generate trajectories and distills chain-of-thought reasoning from LLMs to smaller models. Lumos [10] converts ground-truth reasoning steps from existing benchmarks into tool-usage trajectories. TASKBENCH [24] samples trajectories from pre-defined graphs. MAT-AGENT [5] scales up trajectory tuning for open VLM controllers with a diverse tool suite and synthetic tasks.

2.3 Reflective and Self-Correcting Agents↩︎

The concept of reflection—prompting a model to critique its own outputs—has been widely explored in LLMs. Reflexion [25] and similar frameworks [26], [27] use verbal feedback and episodic memory to improve performance over multiple trials at inference time. However, these methods typically treat reflection as a frozen prompting strategy or rely on external scalar feedback, without optimizing the reflection generation process itself. In contrast, ReGRPO explicitly learns how to generate a diagnostic reflection and how that reflection guides the next corrective tool call under an explicit trigger mechanism. Rather than relying on inference-time trial-and-error alone, ReGRPO trains the model to produce grounded diagnostics that enable successful recovery, internalizing the correction loop into the policy. We evaluate this approach on GTA and GAIA to assess effectiveness and sample efficiency in tool-using settings.

3 Method↩︎

Existing training strategies for tool-using agents have two key issues. First, supervised trajectories rarely include recovery steps [5]. As a result, the model learns only successful traces. A small mistake can then cascade without guidance on how to fix it. For example, in receipt QA, SFT teaches the correct object_loc followed by OCR. At test time, a slight layout shift may lead to a nearby crop and empty OCR, but the training data provides no corrective signal (e.g., expand the box or re-localize). Second, standard RL provides weak localization. A multi-step document QA run can fail after a wrong crop, then an empty OCR, then an incorrect answer. A final reward of 0 does not reveal which step was wrong or how to correct it. This ambiguity makes learning slow and unstable.

We introduce ReGRPO (Reflection-Augmented Group Relative Policy Optimization), which explicitly learns how to generate diagnostic reflection and how reflection guides the next corrective tool call in local trajectories. As shown in Figure 2, ReGRPO follows three principles. First, it uses a Structured Reflective Data Engine that converts execution failures into grounded diagnostic triplets. Second, it applies ReGRPO to optimize the generation of reflection steps that lead to successful corrections. Third, it uses a Zero-Verifier Inference Stage with single-path execution and no external verifier calls. We evaluate the resulting agent on GTA and GAIA benchmarks to assess overall tool-use performance.

The remainder of this section presents the problem setup (Sec. 3.1), the Structured Reflective Data Engine (Sec. 3.2), ReGRPO training (Sec. 3.3), and the inference strategy (Sec. 3.5).

3.1 Problem Setup↩︎

A task is given by \((Q, F)\), where \(Q\) is a user query and \(F\) is a set of files or images. At step \(i\), the agent observes history \(h_i = \{(t_j, c_j, o_j)\}_{j=1}^{i-1}\). Standard agents predict an action \(a_i = (t_i, c_i)\), where \(t_i\) is a thought and \(c_i\) is a tool call.

In ReGRPO, the action space includes an additional reflection step \(z_i\). When an action \(a_i^{(0)}\) fails and produces observation \(o_i^{(0)}\), the agent may generate reflection \(z_i\) before attempting correction \(a_i^{(1)}\). The local trajectory segment is \[\tau_i = \begin{cases} (a_i^{(0)}, o_i^{(0)}) & \text{if success} \\ (a_i^{(0)}, o_i^{(0)}, z_i, a_i^{(1)}, o_i^{(1)}) & \text{if reflection triggered} \end{cases} \label{eq:traj}\tag{1}\] The goal is to learn a policy \(\pi_\theta\) that both acts and reflects to self-correct when necessary.

3.2 Structured Reflective Data Engine↩︎

Existing trajectory cloning methods and outcome-based reward models provide only coarse supervision in multimodal environments. When a trajectory fails, a scalar penalty does not reveal which step or parameter caused the error, so the model cannot learn how to repair it. To address this, we introduce a Structured Reflective Data Engine that converts failures into explicit causal evidence. Instead of treating failed actions as generic negatives—which can worsen off-policy shifts—we reformulate them as instructional “Error-Reflection-Correction” trajectories, giving the agent concrete recovery supervision from the outset.

Failure Induction. To teach recovery, the model must observe concrete failure states rather than only successful traces. We therefore construct initial failures that are realistic yet recoverable, so the agent can learn how errors arise and how to fix them. Specifically, starting from a ground-truth step \(a_i^*\) (from MM-Traj [5]), we synthesize plausible “near-miss” actions \(a_i^{fail}\) by perturbing tool choices or arguments (e.g., shifting a bounding box, selecting an adjacent table column, or calling a mismatched tool). We then execute \(a_i^{fail}\) in the sandbox to obtain real failure observations \(o_i^{fail}\) (e.g., API exceptions, empty OCR outputs, or irrelevant crops). These executed failures provide grounded error signals that the Reflective Data Engine links to diagnostic reflections and corrections. We organize the resulting annotations using a Reflection-of-Thought (RoT) reflection \(z_i\). Each annotation is a RoT triplet \((a_i^{fail}, o_i^{fail}, z_i)\) paired with the corrected action \(a_i^*\), which together explicitly encode the failure, structured reflection, and recovery target.

Reflection Annotation. To bridge the causal gap between the faulty action \(a_i^{fail}\) and the failure observation \(o_i^{fail}\), we use a teacher vision-language model (GPT-4o by default) to generate a structured RoT reflection \(z_i\), conditioned on the failure context \((h_i, a_i^{fail}, o_i^{fail})\). To prevent the agent from generating free-form, hallucinated excuses, \(z_i\) is strictly constrained to a triplet schema:

• ErrorType: A categorical diagnosis of the failure (e.g., ToolMismatch, ArgInvalid, GroundingDrift, InfoInsufficient).

• Evidence: A mandatory reference to the visual or textual observation that triggered the error. For instance, rather than a generic “the tool failed”, the model must explicitly ground its reasoning: “The OCR output for the specific region #box1 returned empty text, indicating an incorrect crop.”

• FixPlan: A concrete, actionable natural language strategy to correct the error and reach the target state (e.g., “Expand the bounding box slightly to cover the text” or “Switch to a visual localization tool”).

Correction Pairing and Internalization. Finally, the original ground-truth action \(a_i^*\) is appended as the corrected action, closing the causal loop. This pipeline converts the MM-Traj dataset [5] into a supervised corpus of augmented trajectories: \(\tau_i^{reflective} = (x_i, a_i^{fail}, o_i^{fail}, z_i, a_i^*)\). During supervised fine-tuning (SFT), the model is forced to maximize the conditional likelihood \(P(z_i, a_i^* \mid x_i, a_i^{fail}, o_i^{fail})\). Consequently, the agent internalizes the multimodal error diagnostic capability at the parameter level, enabling subsequent training-time exploration over reflective recoveries in ReGRPO.

3.3 Reflection-Augmented Group Relative Policy Optimization↩︎

Dynamic Local Trajectory Formulation. For a given state context \(x_i\), rather than scoring atomic actions, we form a group of \(K\) candidate local trajectories \(\{\tau_i^{(k)}\}_{k=1}^K\) and score each under the current policy \(\pi_\theta\). The structure of each candidate trajectory depends on the intermediate environmental feedback: a candidate begins with an initial action \(a_i^{(0)}\) and its observation \(o_i^{(0)}\). If the execution succeeds, the trajectory terminates early: \(\tau_i = (a_i^{(0)}, o_i^{(0)})\). Otherwise, if \(o_i^{(0)}\) indicates a hard failure (e.g., execution error, empty visual crop) or low task–observation consistency, the policy generates a diagnostic reflection \(z_i\) and a corrective action \(a_i^{(1)}\), forming the recovery segment in 1 .

Reflection-Aware Process Reward. To encourage meaningful exploration while penalizing infinite loops or verbosity, we define a composite reward function \(R(\tau)\) that balances task success with execution efficiency: \[R(\tau) = \lambda_{\text{exec}}\,\mathbf{1}\{\text{success}\} - \eta\,C(\tau) + \lambda_{\text{val}}\,V(x_i, \tau). \label{eq:reward}\tag{2}\] Here, \(\mathbf{1}\{\text{success}\}\) is deterministic environment feedback indicating task completion, and \(C(\tau)\) penalizes unnecessary reflection length. The first two terms already form a complete verifier-free objective. \(V(\cdot)\) is an optional training-only verifier value, computed deterministically from the active record’s RoT metadata (no model is queried inside the RL loop): it can be enabled with \(\lambda_{\text{val}}>0\) for extra stabilization, or disabled with \(\lambda_{\text{val}}=0\) without changing the deployment algorithm. Default coefficients in our reported setting are \(\lambda_{\text{exec}}=1.0\), \(\lambda_{\text{val}}=0.3\), and \(\eta=0.1\).

The term \(C(\tau)\) introduces a reflection cost penalty. If the trajectory invokes the reflection step \(z_i\), \(C(\tau)\) is proportional to the token length of \(z_i\); otherwise, for a one-shot success, \(C(\tau) = 0\). The penalty coefficient \(\eta\) forces the agent to reflect only when strictly necessary, ensuring that the expected reward gain from a successful recovery strictly outweighs the penalty of generating additional reasoning tokens.

Optional training-time verifier design. When enabled, the verifier value \(V(x_i,\tau)\) is computed deterministically from each candidate and the active record’s RoT metadata; no GPT-4o (or any LLM) is queried inside the RL loop. We derive three subscores in \([0,1]\) by signature matching and a grounded-reflection check rather than by model scoring:

• Plan validity \(s_p\): \(s_p=1\) iff the candidate’s normalized primary tool and first argument match the stored corrected-action signature, else \(0\).

• Answer consistency \(s_a\): \(s_a=1\) iff the candidate’s replay succeeds (its terminal action agrees with the group’s correct answer), else \(0\).

• Grounding \(s_g\): \(s_g=1\) iff the candidate carries a reflection whose evidence is text-grounded in the stored failure observation and \(s_p>0\), else \(0\). There is intentionally no fallback to \(s_p\) for reflection-less candidates, so \(V\) rewards grounded reflection above a bare plan repair.

The overall score is the weighted sum \(V = w_a s_a + w_g s_g + w_p s_p\), with each subscore clamped to \([0,1]\) and weights set to emphasize grounding (\(w_g \ge w_a, w_p\)); we use \((w_a,w_g,w_p)=(0.25,0.50,0.25)\) by default. Because \(V\) is a function of the candidate and the metadata, it is used only as a training-time reward-shaping signal and never as a deployment-time call. A GPT-4o teacher is used only offline, to synthesize the RoT reflections in the data engine (Sec. 3.2); it is never queried during RL or at inference. Full coefficient ranges, the deterministic subscore definitions, and the selection protocol are provided in Appendix 8.

3.4 Training Objective↩︎

Structured Reflective Data in Supervised Fine-Tuning (warm start). We first teach the model to diagnose and correct failures with explicit supervision, so Reinforcement Learning (RL) can focus on how reflection should be written under explicit trigger gating rather than learning recovery from scratch. Given the failure context \((x_i, a_i^{fail}, o_i^{fail})\), we maximize the likelihood of the structured reflection and corrected action: \[\mathcal{L}_{\mathrm{SFT}} = -\mathbb{E}_{(x_i,a_i^{fail},o_i^{fail},z_i,a_i^*)} \left[ \log P_\theta(z_i, a_i^* \mid x_i, a_i^{fail}, o_i^{fail}) \right].\] This stage teaches the model what to diagnose and how to fix errors, providing a strong initialization for subsequent RL.

Structured Reflective Data in Policy Optimization. SFT on RoT data teaches the model to fix common tool-call errors (e.g., crop shift, wrong region, mismatched tool arguments), but real-world failures are more diverse. Therefore, to generalize beyond curated patterns, the agent must self-explore to discover new failure modes and recovery strategies. This motivates an RL stage to expand coverage and improve robustness.

However, PPO [28] and DPO [29] optimize actions or preference pairs without modeling reflection as a decision variable. In long-horizon tool use, a scalar success/failure signal does not indicate which step failed or what correction would fix the trajectory. Moreover, DPO relies on preference data that is difficult to collect for multi-step tool executions, and PPO can be sample-inefficient under sparse rewards. In contrast, GRPO [30] optimizes relative rewards within a group of sampled trajectories, producing lower-variance advantages that are well suited to sparse, delayed rewards. By comparing recoveries against nearby failures, GRPO yields more informative gradients than absolute-reward optimization.

As illustrated in Figure 3, ReGRPO extends GRPO by making reflection \(z_i\) part of the optimized trajectory. This ties the advantage signal directly to diagnostic tokens, so the policy learns which reflections are useful and how much reflection is worth generating. Consequently, reflection quality and brevity become explicitly optimized rather than a fixed prompting heuristic.

Reflection-Driven Advantage and Optimization. Within the sampled group, we compute the baseline as the mean reward \(\bar{R}_i = \frac{1}{K}\sum_{k=1}^K R(\tau_i^{(k)})\), and extract the advantage for each trajectory: \(A_i^{(k)} = R(\tau_i^{(k)}) - \bar{R}_i\).

For two trajectories from the same state context, an unrecovered failure \(\tau^{-}\) and a recovered trajectory \(\tau^{+}\), the reward gap is \(\Delta R = R(\tau^{+}) - R(\tau^{-})\). When recovery succeeds and reflection remains concise, \(\Delta R\) is typically positive, so \(A(\tau^{+})>A(\tau^{-})\) within the same sampled group. Importantly, this remains true in the verifier-free setting (\(\lambda_{\text{val}}=0\)), where improvements come entirely from reflection-conditioned recovery and reflection-cost control.

The ReGRPO objective is optimized as: \[\mathcal{L}_{\mathrm{ReGRPO}} = -\mathbb{E}_{x_i, \tau \sim \pi_\theta} \left[ \frac{1}{K} \sum_{k=1}^K A_i^{(k)} \log \pi_\theta(\tau_i^{(k)}\mid x_i) \right] + \beta\,\mathbb{D}_{\mathrm{KL}}\!\big(\pi_{\theta}\|\pi_{\text{ref}}\big)\] Since the generation probability of structured reflection \(\log \pi_\theta(z_i \mid x_i, a_i^{(0)}, o_i^{(0)})\) is factorized within the trajectory likelihood \(\log \pi_\theta(\tau_i^{(k)}\mid x_i)\), relative advantages directly scale gradients on reflection tokens. ReGRPO therefore uses the standard GRPO optimizer and advantage estimator; the contribution is the structured reflection trajectory design and training/deployment protocol, not a new value estimator. Detailed quantitative results are presented in Sec. 4, and extended diagnostics are reported in the appendix.

3.5 Zero-Verifier Inference Stage↩︎

At deployment, we use single-path inference without external verifier calls. A deterministic gate decides whether the policy should open one local reflection-correction block. Tool outputs are first normalized into a canonical schema \[\hat{o}_i = \{\texttt{status},\texttt{payload},\texttt{meta}\},\] where status stores backend error codes and payload stores normalized content. We then use a minimal trigger \[g_i = \mathbf{1}\{\texttt{ToolError}(\hat{o}_i) \lor \texttt{EmptyObs}(\hat{o}_i) \lor u_i < \kappa_i\},\] with policy confidence \[u_i = \exp\!\left(\frac{1}{|a_i^{(0)}|}\sum_j \log \pi_\theta(a_{i,j}^{(0)}\mid x_i,h_i)\right).\] The first two terms capture explicit runtime failures. The confidence term is a lightweight uncertainty proxy: it measures token-level confidence for the predicted action sequence. We therefore use it only as a trigger heuristic for potential silent failures. To avoid per-tool offline calibration, we use an online adaptive threshold computed from the current trajectory: \[\kappa_i = \frac{1}{\max(1,i-1)}\sum_{j=1}^{i-1} u_j.\] For \(i=1\), we disable the confidence trigger and rely on hard-failure checks only. If \(g_i=0\), inference continues with the next standard action. If \(g_i=1\), the policy executes one local block \(a_i^{(0)}\rightarrow o_i^{(0)}\rightarrow z_i\rightarrow a_i^{(1)}\rightarrow o_i^{(1)}\), with at most one reflection-correction block per step. This design keeps inference deterministic and lightweight, while reducing manual feature engineering and per-tool tuning.

Pseudo-code for this trigger is provided in the Appendix.

4 Experiments↩︎

4.1 Benchmarks and Metrics↩︎

We evaluate ReGRPO on two multimodal tool-use benchmarks in the MAT-AGENT setting [5], where agents must reason over images and documents with real tools.

GTA Dataset. GTA [7] contains 229 tasks paired with 252 images. Each task requires 2–8 tool-use steps (typically 2–4) and tests visual perception, local operations (e.g., reading receipts or charts), and short reasoning chains over screenshots and UI-like images.

GAIA Dataset. GAIA [8] is a document-centric benchmark with 446 tasks over 109 files in PPTX, PDF, and XLSX formats. Tasks are grouped into three difficulty levels and often require multiple tool calls for document understanding, web navigation, logical reasoning, and summarization.

Metrics. We report standard accuracy metrics: AnsAcc (answer accuracy), ToolAcc (tool-call validity), and CodeExec (execution success rate). Unless noted otherwise, all main-text comparisons use zero-verifier deployment, with no external verifier calls at test time.

4.2 Baselines↩︎

We compare ReGRPO with both closed-source and open-source controllers.

Closed-source agents. We report GPT-4/4o-based results from prior work, including Lego Agent and Warm-up Agent [7], [8], and HF Agents powered by GPT-4o and GPT-4o-mini. These models provide strong proprietary references.

Open-source agents. We include HF Agents based on LLaVA-NeXT-8B, InternVL2-8B, MiniCPM-V-8.5B, and Qwen2-VL-7B. We also evaluate MAT-AGENT (T3-Agent) with MAT-MiniCPM-V-8.5B and MAT-Qwen2-VL-7B, and SPORT Agent with a tuned Qwen2-VL-7B controller. For fair internal comparisons, all ablations use the same Qwen2-VL-7B controller and tool suite.

Our model. We use the same Qwen2-VL-7B backbone and toolset as SPORT [6] to isolate the effect of our method. We compare controlled variants under matched training settings (same total updates, rollout budget, and token budget): (1) MAT-AGENT: Fine-tuned on MM-Traj without RL; (2) +RoT SFT: SFT with structured reflective data only; (3) +Optional Verifier Distill: adds verifier-aware distillation but no RL; (4) GRPO-only: GRPO on actions only (no reflection tokens); (5) GRPO + Free-form Reflection: allows unstructured reflection text without schema constraints; (6) ReGRPO core (default, verifier-free RL): structured reflection with \(\lambda_{\text{val}}=0\) during RL; (7) ReGRPO + optional verifier reward: additive shaping with the deterministic, metadata-derived verifier value (\(\lambda_{\text{val}}>0\)). All variants use Single-Path, Zero-Verifier Inference at test time.

4.3 Implementation Details↩︎

Model and optimization. We use Qwen2-VL-7B as the controller. The vision encoder and visual token compressor are frozen. We fine-tune the language model with LoRA [31] (rank 32), applied to query, key, and value projections in all self-attention layers. We optimize with AdamW and cosine learning-rate decay, using a base learning rate of \(1.0\times10^{-6}\) and batch size 2 per device.

Training stages. Our default pipeline has two mandatory stages: (1) SFT warm start on our Structured Reflective Data (triplets); (2) ReGRPO process RL on a mixture of offline groups and online self-exploration groups. Optionally, we insert verifier-aware distillation between (1) and (2), where the controller predicts the deterministic, metadata-derived verifier subscores for better calibration and more stable initialization. ReGRPO uses the reflection-aware reward in Eq. 2 , with a reflection-cost penalty \(\eta=0.1\). By default, we set \(\lambda_{\text{val}}=0\) (verifier-free RL); runs with \(\lambda_{\text{val}}>0\) using the deterministic, metadata-derived verifier value are reported as additive variants. For controlled comparisons, variants with and without distillation use identical optimizer settings and matched update steps to avoid gains from extra training budget.

4.4 Quantitative Results↩︎

Table 1 summarizes results on GTA and GAIA [7], [8]. Under the same single-path, zero-verifier setup, default ReGRPO (\(\lambda_{\text{val}}=0\)) gives the strongest results among the compared open-source controllers. On GTA, it reaches \(76.35\) ToolAcc and \(67.66\) AnsAcc, improving over SPORT [6] by \(+3.94\) ToolAcc and \(+7.40\) AnsAcc. The larger AnsAcc gain suggests that reflection helps end-to-end reasoning beyond tool-call validity alone. On GAIA, default ReGRPO raises overall AnsAcc to \(23.35\) (\(+2.74\) over SPORT), indicating stronger results on document-centric multi-step tasks even without verifier reward. Adding the optional teacher-derived verifier reward further improves performance to \(68.49/24.01\) GTA/GAIA AnsAcc in Table 2, while the main gains are already achieved in verifier-free training and deployment. Across both datasets, the results support our hypothesis that structured reflections improve tool-grounded reasoning. Multi-seed statistics and extended analyses are provided in the appendix.

4.5 Ablation Studies↩︎

Table 2 reports a controlled ablation in which we progressively add reflective supervision and reflection-aware optimization.

Contribution coverage. Our experimental design covers these method components: (1) the Structured Reflective Data Engine is evaluated by MAT-AGENT \(\rightarrow\) +RoT SFT and the data-format comparison in Figure 4; (2) the GRPO-based reflection protocol is demonstrated by comparing GRPO-only and free-form reflection against structured ReGRPO under the same settings; (3) deterministic zero-verifier trigger gating is assessed under single-path, zero-verifier deployment, with extended trigger diagnostics in the appendix.

Baseline and reflective SFT gains. Starting from MAT-AGENT (MM-Traj, no RL), performance is \(53.85\) GTA AnsAcc and \(16.97\) GAIA AnsAcc. Replacing SFT data with RoT (+RoT SFT) raises GTA AnsAcc to \(58.59\) and GAIA AnsAcc to \(19.03\) (\(+4.74\) GTA / \(+2.06\) GAIA), showing that explicit Error-Reflection-Correction supervision improves trajectory quality before RL.

Optional verifier distillation contributes modestly. Adding optional verifier distillation on top of RoT SFT further improves performance to \(59.72\) GTA AnsAcc and \(19.84\) GAIA AnsAcc (an additional \(+1.13\) GTA / \(+0.81\) GAIA over +RoT SFT). This suggests the deterministic verifier subscores help intermediate decisions, but alone they do not close the gap to RL-based methods.

Action-only RL vs. reflection-aware variants. GRPO-only (without RoT reflection tokens in policy optimization) reaches \(64.51\) GTA AnsAcc and \(18.92\) GAIA AnsAcc, indicating that RL exploration is effective on GTA but more limited on GAIA. Allowing free-form reflection improves results to \(65.34\) GTA AnsAcc and \(21.38\) GAIA AnsAcc (\(+0.83\) GTA / \(+2.46\) GAIA over GRPO-only), suggesting reflection text helps, while unstructured reflection remains less efficient and less consistent.

Effect of structured ReGRPO and optional verifier reward. Switching from free-form reflection to structured ReGRPO with \(\lambda_{\text{val}}=0\) yields \(67.66\) GTA AnsAcc and \(23.35\) GAIA AnsAcc (\(+2.32\) GTA / \(+1.97\) GAIA), showing that schema-constrained reflection and reflection-correction coupling add gains even without verifier reward in RL. Adding the optional teacher-derived verifier reward further improves to \(68.49\) GTA AnsAcc and \(24.01\) GAIA AnsAcc (\(+0.83\) GTA / \(+0.66\) GAIA over ReGRPO core), indicating that verifier reward is useful but not essential.

Key insight. The ablation shows complementary effects: (1) RoT data provides a strong starting point, (2) structured reflection-aware RL adds consistent gains on top of that initialization, and (3) verifier signals provide modest additive shaping rather than core capability. ReGRPO therefore remains effective in a verifier-free default setting during both training (\(\lambda_{\text{val}}=0\)) and inference (zero-verifier single-path execution). We further verify that ReGRPO preserves the base model’s VQA ability (Appendix Sec. 11).

4.6 Qualitative Analysis↩︎

Figure 4 presents three views of the mechanism, instantiated on a verbatim synthesized RoT record (a GroundingDrift case). Panel (a) shows that RoT extends the MAT SFT format with explicit reflective fields (a reflection triplet and a corrected_action), so supervision covers failure diagnosis and correction rather than tool calls alone. Panel (b) gives case-level evidence aligned with Sec. 3.5: the tool returns a valid but off-target description (it answers about the cutting board, not the smoothie), so no explicit error is raised, and low confidence (\(u_i<\kappa_i\)) triggers a structured reflection whose schema (ErrorType, Evidence, FixPlan) re-grounds the query rather than adding generic free-form text. Panel (c) shows the same mechanism as paths: the baseline route estimates from off-target items, while ReGRPO opens exactly one local block at the uncertain step and rewrites the action to re-ground on the target object. The full record is given in Appendix Table 12.

5 Conclusion↩︎

We presented ReGRPO, a reflection-augmented framework for training tool-using vision–language agents. ReGRPO builds a Structured Reflective Data Engine and applies a GRPO-based reflection protocol to optimize reflection and action generation jointly in tool-using trajectories. Our method keeps the standard GRPO optimizer, with the main design changes in structured reflection representation and the training/deployment protocol.

On the GTA and GAIA benchmarks, our approach consistently outperforms action-level and SFT-only baselines under the same backbone/tool setting. The ablation results show complementary gains: RoT SFT provides a strong foundation, structured RL contributes further improvements, and optional verifier distillation/reward adds further but smaller gains. Overall, ReGRPO improves answer accuracy and tool correctness over MAT-AGENT and SPORT while using the same VLM backbone and tool suite.

Acknowledgements.↩︎

This project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award No: MOE-T2EP20124-0012).

6 Algorithmic Details↩︎

In this section we provide pseudo code and implementation details for Reflection-Augmented Group Relative Policy Optimization (ReGRPO).

6.1 ReGRPO Training↩︎

We assume a dataset of task states. For each state \(s_i\), we sample a group of trajectories and optimize a verifier-aware reward during training. The verifier term is used only in training; inference is strictly zero-verifier.

6.2 Reward Coefficients and Group Size↩︎

Table 3 lists the default coefficients used in our reported setting.

6.3 Inference Trigger Implementation↩︎

We use a deterministic and lightweight trigger that avoids per-tool feature engineering. For each step, tool output is normalized as \[\hat{o}_i = \{\texttt{status},\texttt{payload},\texttt{meta}\},\] where status records runtime errors and payload stores normalized content. The trigger is \[g_i = \mathbf{1}\{\texttt{ToolError}(\hat{o}_i) \lor \texttt{EmptyObs}(\hat{o}_i) \lor u_i < \kappa_i\}.\] Hard failures are captured by ToolError/EmptyObs; silent failures are captured by low policy confidence \[u_i = \exp\!\left(\frac{1}{|a_i^{(0)}|}\sum_j \log \pi_\theta(a_{i,j}^{(0)}\mid x_i,h_i)\right).\] To avoid offline calibration and per-tool thresholds, we use an online adaptive threshold \[\kappa_i = \frac{1}{\max(1,i-1)}\sum_{j=1}^{i-1} u_j.\] For \(i=1\), confidence-based triggering is disabled and only hard-failure checks are used.

This design keeps deployment deterministic, simple, and reproducible while reducing manual tuning.

7 Structured Reflective Data Engine Statistics↩︎

Table 4 reports corpus statistics over the synthesized Reflection-on-Thought (RoT) data (\(16{,}552\) records). Each record is labeled with one of the four ErrorType categories used throughout the paper (Sec. 3.2): ArgInvalid, ToolMismatch, InfoInsufficient, and GroundingDrift. This four-way taxonomy is the reflection label set: ArgInvalid dominates (\(45.4\%\)), reflecting that corrupted-argument failures (wrong path/index/span) are the most frequent recoverable error in the source trajectories, followed by tool-capability mismatches (\(24.7\%\)), insufficient-information queries (\(19.7\%\)), and grounding drift (\(10.1\%\)). The corpus is multi-modal: \(37.0\%\) of records carry an image, while the remaining \(63.0\%\) are text-only.

8 Verifier Subscores and Offline Teacher↩︎

We stress that no LLM verifier is queried inside the RL loop. The verifier value \(V\) used by the \(\lambda_{\text{val}} V\) reward term is computed deterministically from each candidate and the active record’s RoT metadata, so \(V\) is a function of the candidate, not of any network or external model call: \[V = w_a s_a + w_g s_g + w_p s_p, \quad w_g \ge w_a, w_p, \quad s_a,s_g,s_p \in [0,1],\] with default weights \((w_a, w_g, w_p) = (0.25, 0.50, 0.25)\). The three subscores are derived as follows (see Fig. 7):

• \(s_p\) (plan validity): \(s_p = 1\) iff the candidate’s normalized primary tool and first argument match the stored corrected_action signature, else \(0\).

• \(s_a\) (answer consistency): \(s_a = 1\) iff the candidate’s replay succeeds (its terminal action agrees with the group’s correct answer), else \(0\).

• \(s_g\) (grounding): \(s_g = 1\) iff the candidate carries a reflection whose evidence is text-grounded in the stored failed_observation and \(s_p > 0\), else \(0\). There is intentionally no fallback to \(s_p\) for reflection-less candidates, so \(V\) rewards grounded reflection above bare plan repair.

A GPT-4o teacher is used only offline, to synthesize the RoT reflections in the data engine (Sec. 10); it is never called during RL training or at inference.

None

9 Tool Suite↩︎

We use the standard MAT-AGENT tool suite:

• Web search: searchinformation, visit, webqa.

• Image QA: image_qa.

• File inspector: pdf_span, table_reader.

• Object localization: object_loc.

• Python sandbox: python_exec.

10 Prompt Templates↩︎

10.1 Reflection Generation Prompt↩︎

The Reflection-on-Thought (RoT) corpus is synthesized offline by a teacher LLM (GPT-4o). For each source step, the teacher is asked to produce, under a single requested ErrorType, one realistic near-miss failed action, one faithful failed observation in the target tool’s native output format, and one structured reflection triple. The system prompt fixes the contract and enforces strict JSON output:

The user message is a JSON payload that supplies the task, the source step, the requested_error_type drawn from the four-way taxonomy \(\{\)ArgInvalid, ToolMismatch, InfoInsufficient, GroundingDrift\(\}\), a per-error-type guidance string, a per-tool failure-observation template, the history before the step, and the (teacher-only) correct action. It also fixes the exact output schema the teacher must emit:

The per-error-type guidance instantiates each label concretely, e.g.GroundingDrift “shift the referenced region, object, crop, or visual target while preserving the tool shape”; ToolMismatch “replace the tool with a different, unsuitable tool for the same intent”; ArgInvalid “corrupt an argument such as a path, page, bbox, object name, query, size, or format”; and InfoInsufficient “drop or underspecify necessary context so the tool cannot retrieve enough information.” Every emitted record passes a deterministic validator that, among other checks, requires reflection.evidence to be text-grounded in failed_observation (a literal token overlap heuristic) and rejects any record whose error_type lies outside the taxonomy.

11 Base VQA Ability Is Preserved↩︎

A natural concern is whether reflection-augmented RL erodes the controller’s underlying vision–language ability. Evaluating the base model and the ReGRPO-trained model on MMBench, we find that ReGRPO preserves base VQA ability (\(85.0 \to 84.25\), a \(0.75\)-point change), so learning structured reflection does not come at the cost of general VQA competence.

12 Qualitative Study and Case Studies↩︎

We present trajectory-level case studies under the same inference setup as Sec. 3.5: inference is single-path and zero-verifier, and at each step we apply the deterministic gate \(g_i=\mathbf{1}\{\texttt{ToolError}(\hat{o}_i) \lor \texttt{EmptyObs}(\hat{o}_i) \lor (i>1 \land u_i<\kappa_i)\}\), allowing at most one local reflection-correction block per step. All trajectories and records are copied verbatim from the ReGRPO evaluation caches and the Structured Reflective Data Engine corpus; no trajectory is paraphrased or synthesized, and long fields are truncated with “[…]” for readability only.

12.1 Success Trajectories↩︎

We first show two correct end-to-end trajectories from the GTA evaluation cache. Correctness is checked with GTA’s official substring rule (for every gold list, at least one gold string is a case-insensitive substring of the final answer). Trajectories use the runtime MAT-AGENT tool names (visualizer, ask_search_agent), which correspond respectively to the image-QA and web-search tools.

12.1.0.1 Case A1: GTA multi-step tool use — constrained menu selection.

Task: “Choose only one dish in Salad and one dish in Sandwich, which combination can be no more than 10.9 yuan in total?”
Input: image_318.jpg (a menu, Figure 8). Gold: {Toast, Garden Green Salad}.

12.1.0.2 Case A2: GTA search \(+\) reason — image \(\rightarrow\) entity \(\rightarrow\) web fact.

Task: “Who is the CEO of this company?”
Input: image_417.jpg (an iPhone 15 Pro, Figure 9). Gold: {Tim Cook}.

12.2 Failure Modes and Reflective Recovery↩︎

The four records below are copied verbatim from the Structured Reflective Data Engine corpus (Sec. 3.2) and cover the four ErrorType categories used throughout the paper: ArgInvalid, ToolMismatch, InfoInsufficient, and GroundingDrift. For one error type each, the record shows the task, the failed (perturbed) action \(a_i^{fail}\), the resulting failure observation \(o_i^{fail}\), the synthesized structured Reflection-on-Thought \(z_i\) (ErrorType / Evidence / FixPlan, the schema of Sec. 3.2), and the corrected action \(a_i^{*}\). During supervised fine-tuning the reflection is emitted in-context as a Reflection: block immediately before the corrected action, so the model maximizes \(P(z_i, a_i^{*}\mid x_i, a_i^{fail}, o_i^{fail})\).

References↩︎

---

1. Corresponding author.↩︎