1 Introduction↩︎

Multimodal large language models (MLLMs) have made remarkable progress in visual understanding and reasoning, largely powered by the text-based chain-of-thought (CoT) paradigm that allows models to decompose complex problems step by step  [1]–[4]. Despite its effectiveness, this text-centric formulation treats visual input as a static, one-shot context: the image is consumed at the outset, and all subsequent intermediate reasoning unfolds entirely in the linguistic domain. This design introduces a fundamental semantic gap between the richness of perceptual information and the discrete, symbolic nature of linguistic thought. Human cognition, by contrast, naturally extends beyond language: we sketch diagrams, zoom into regions of interest, and spatially rotate objects, effectively using vision as a dynamic mental scratchpad rather than a frozen snapshot. Recent advances in multimodal large language models (MLLMs) have begun to mirror this tendency, giving rise to a paradigm broadly termed think with images. Under this paradigm, multimodal agents iteratively invoke visual tools to actively probe, manipulate, and scrutinize visual inputs at each reasoning step [5]–[12]. This tool-augmented paradigm reframes tasks such as visual search, chart analysis, and interactive perception as long-horizon decision problems, where the central challenge is no longer only how to reason over pixels, but how to accumulate stable, reusable tool-use capabilities across diverse tasks.

Reinforcement learning has become the dominant paradigm for training multimodal tool-use agents, with outcome rewards, process rewards, and tool-use rewards all demonstrating the ability to steer a policy toward more effective tool routing under GRPO-style [13] optimization. Yet representative methods, such as DeepEyes [10], OpenThinkIMG [9], VTool-R1 [11], and CodeVision [12], share a common design assumption: a completed trajectory is valuable only as a source of scalar rewards, keeping the focus squarely on reward shaping rather than on what the trajectory itself encodes. As a consequence, recurring tool-use patterns are absorbed only implicitly through gradient updates and must be rediscovered from scratch on every new task. The root technical failure is therefore representational: successful trajectories are consumed as reward signals rather than distilled into reusable, structured knowledge, making long-horizon capability accumulation both unstable and sample-inefficient.

In the text domain, a complementary line of work improves model performance by leveraging past experience, rather than optimizing the model itself. Systems such as Reflexion [14], Voyager [15], and ExpeL [16] show that long-horizon behavior improves when agents can store traces, reflect on failures, and extract reusable experience. More recent methods distill trajectories into evolving cheatsheets [17], procedural memories [18], or hierarchical knowledge bases [19] for test-time retrieval. While appealing, these approaches do not establish a closed feedback loop between experience and policy. Memory construction and policy improvement remain decoupled, so stronger policies do not automatically yield better experience, and improved experience does not in turn enhance subsequent policy behavior.

For multimodal tool-use agents, the challenge is greater than in text-only settings. These agents act over heterogeneous visual bottlenecks, and each tool action changes the evidence state for subsequent decisions. As a result, useful experience is not captured solely by whether a trajectory succeeds, but by the structured interaction patterns it reveals among perception, tool use, and reasoning. The flat memory structure is insufficient in this setting, because multimodal experience spans both fine-grained local execution and higher-level task organization. A single-level memory cannot clearly separate these two forms of reusable knowledge.

To address these limitations, we propose SPyCE(Skill-Policy Co-evolution), a framework that closes the loop between skill and policy for multimodal tool-use agents. As illustrated in Figure 1, SPyCE distills rollout trajectories into a hierarchical skill library that co-evolves with the policy throughout reinforcement learning. The skill abstraction consists of two layers. Execution skills encode reusable local visual operations as condition–action–effect triples, making each operation explicit in terms of when it should be applied, how it is executed, and what improvement in evidence it is expected to produce. Workflow skills encode coarse-grained bottleneck descriptions and workflow-level priors for organizing multi-step reasoning. During execution, the policy conditions on hierarchically retrieved skills to plan and act under the current visual bottleneck. High-quality rollouts periodically refresh both skill libraries through a merge-or-add rule that suppresses noise and redundancy. This closed loop distinguishes our framework from prior work: a stronger policy produces higher-quality rollouts, better rollouts yield better skills, and better skills in turn provide stronger priors for subsequent policy updates.

We conduct comprehensive experiments on eight benchmarks including agentic reasoning, general multimodal reasoning, and visual search. The results show that SPyCE consistently outperforms all RL-based and memory-based baselines. Compared with RL-based methods, our framework demonstrates the importance of introducing skill abstractions into policy optimization. Compared with memory-based methods, it further highlights the advantage of co-evolving policy and skills, which enables skills not only to be accumulated but also to be continuously transformed into stronger future behavior. Ablation studies further confirm that both levels of the hierarchical skill abstraction are necessary and complementary, with workflow skills guiding global multi-step planning, execution skills supporting precise local decisions under specific visual bottlenecks, and online skill evolution further proving essential beyond a static skill library by continually refining skills from newly improved policy rollouts.

Our contributions are summarized as follows:

• We identify a key limitation of existing RL-based multimodal agents: rollout trajectories are largely consumed as scalar reward signals rather than distilled into reusable knowledge. We propose SPyCE, to the best of our knowledge, the first framework to establish an online co-evolution loop between skill and policy during training.

• We introduce a hierarchical skill library for multimodal tool-use agents, in which execution skills capture reusable local visual operations and workflow skills encode high-level procedural priors. Built on top of this abstraction, we develop an online co-evolution mechanism of skill and policy that periodically refreshes both skill libraries from high-quality rollouts during training.

• Extensive experiments on eight benchmarks demonstrate consistent and substantial gains over both RL-based and memory-based baselines. Ablation studies further reveal two essential factors behind the gains of SPyCE: the hierarchical skill design is necessary, with execution skills and workflow skills providing complementary support for local execution and global planning, while online skill-policy co-evolution is indispensable for sustaining continued policy improvement.

2 Related Work↩︎

2.1 Multimodal Reasoning↩︎

MLLMs are rapidly strengthening their vision-language reasoning capabilities. General-purpose models, including InstructBLIP [20], LLaVA [21], Qwen2.5-VL [22], and GPT-4V [23], have substantially improved visual understanding and instruction following. Building on this progress, subsequent work draws inspiration from chain-of-thought reasoning in text-only domains and extends it to multimodal settings. Methods such as Visual ChatGPT [5] and MM-REACT [6] show that introducing explicit intermediate vision-language reasoning can yield measurable gains on challenging tasks. However, these approaches still treat images largely as static inputs: visual evidence is provided once at the beginning and then consumed passively during inference, rather than being actively revisited, updated, or manipulated as reasoning unfolds. Recent work extends this paradigm toward multimodal tool-use agents by enabling models to interact with external tools over dynamically evolving visual evidence. OpenAI o3 [24] marks an important step in this direction, as it can incorporate images directly into the reasoning process, a capability referred to as thinking with images.

More recent systems such as DeepEyes [10], OpenThinkIMG [9], VTool-R1 [11], and CodeVision [12] push this paradigm further by explicitly training policy models to reason with images and use tools across multi-step interactions. However, these methods primarily emphasize improved reward design, while largely overlooking the intrinsic value of interaction trajectories. Rather than being distilled into reusable structured knowledge, trajectories are mostly reduced to scalar signals for policy optimization. By contrast, our framework makes fuller use of interaction trajectories by distilling them into skills and enabling the co-evolution of skills and policy.

2.2 Learning from Experience↩︎

Empowering agents to learn from past interactions has become a promising direction for further improving agent capabilities. Representative systems such as Generative Agents [25], Reflexion [14], Voyager [15], and ExpeL [16] show that long-horizon behavior can benefit from retaining interaction histories, reflecting on failures, and extracting reusable experience. More recent methods make this design more explicit: Dynamic Cheatsheet [17] maintains an evolving cheatsheet for test-time reuse, MemP [18] models procedural memory over multi-step interaction, and Agent KB [19] organizes cross-domain experience into a hierarchical repository. These methods suggest an alternative path for agent evolution beyond parameter updates. However, they are primarily developed for text-dominant settings and rely on external memory construction followed by test-time retrieval, leaving policy update decoupled from experience evolution. By contrast, our work targets multimodal reasoning under heterogeneous visual observations and closes the loop between skills and policy through joint co-evolution during training.

3 Methodology↩︎

We consider multimodal visual reasoning tasks in which an MLLM grounds its reasoning in task-relevant visual evidence by actively acting on and interacting with images through tool use. We first formalize the task setting and present the optimization objective that underlies our framework (Section 3.1). As shown in Figure 2, our framework couples policy optimization with a hierarchical skill library through two stages: (1) hierarchical skill library construction from successful trajectories during policy evolution (Section 3.2); and (2) a skill-policy co-evolution stage, in which retrieved skills guide policy optimization while improved rollouts are continuously distilled back into the skill library (Section 3.3).

3.1 Preliminaries↩︎

Problem Formulation. We formulate multimodal tool-use reasoning as a partially observable Markov decision process (POMDP) [26]. A task instance \(T=(q,\mathcal{I})\) consists of a natural language query \(q\) and an image set \(\mathcal{I}=\{I_1,\dots,I_m\}\). At each step \(t\), the agent receives an observation \(o_t\) and selects a tool action \(a_t\), which produces a new observation and updates the interaction state. This process yields a trajectory \(\tau=[(s_0,a_0,o_0),\dots,(s_T,a_T,o_T)]\), after which the agent outputs a final answer \(\hat{y}\).

Group-based Policy Optimization. We optimize the policy with reinforcement learning by maximizing the expected reward of sampled tool-use trajectories. For each task input \(x_i\), we sample a group of \(N\) rollouts \(\{\tau_i^{(n)}\}_{n=1}^{N} \sim \pi_{\theta_{\mathrm{old}}}(\cdot \mid x_i)\) from the current policy. Each rollout \(\tau_i^{(n)}\) is assigned a scalar reward \(R(\tau_i^{(n)})\), and policy optimization aims to increase the likelihood of rollouts with higher rewards. Two representative instances are GRPO [13] and RLOO [27]. GRPO normalizes each rollout reward within its sampled group: \[A_i = \frac{R_i - \mathrm{mean}(\{R_j\}_{j=1}^{N})}{\mathrm{std}(\{R_j\}_{j=1}^{N}) + \varepsilon},\] where \(R_i = R(\tau_i)\). The corresponding clipped GRPO objective is \[\begin{align} \mathcal{L}_{\mathrm{GRPO}}(\theta) = \mathbb{E}_i \Big[ \min\!\Big( r_i(\theta)A_i,\, \mathrm{clip}\!\big(r_i(\theta),1-\epsilon,1+\epsilon\big)A_i \Big) -\beta D_{\mathrm{KL}}\!\big(\pi_\theta(\cdot\mid x_i)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x_i)\big) \Big]. \end{align}\]

where \(r_i(\theta)=\frac{\pi_\theta(\tau_i\mid x_i)}{\pi_{\theta_{\mathrm{old}}}(\tau_i\mid x_i)}\) is the importance ratio, and \(\pi_{\mathrm{ref}}\) is a reference policy used for KL regularization. RLOO differs from GRPO by replacing the normalized group statistic with a leave-one-out baseline: \(A_i^{\mathrm{RLOO}} = R_i - \frac{1}{N-1}\sum_{j \neq i} R_j.\)

In our method, the reward for each rollout is defined as \[R(\tau)=R_{\mathrm{acc}}(\tau)-\lambda_{\mathrm{call}}\cdot \mathbf{1}\!\left[\mathrm{Succ}(\tau)=1\right]\cdot N_{\mathrm{tool}}(\tau),\] where \(R_{\mathrm{acc}}(\tau)=r(\tau)\in\{0,1\}\) indicates task success, \(\mathrm{Succ}(\tau)\) indicates whether the task is solved successfully, and \(N_{\mathrm{tool}}(\tau)\) denotes the number of tool calls. Applying the tool-use penalty only to successful rollouts discourages redundant operations without unnecessarily suppressing exploration in failed ones.

3.2 Hierarchical Skill Library↩︎

During the first stage of policy optimization, we construct the skill library from interaction trajectories to provide reusable guidance for subsequent learning. Multimodal tool-use reasoning requires skills at two complementary granularities: workflow skills for high-level task guidance and execution skills for fine-grained decision support during interaction. Local visual bottlenecks, such as rotated evidence, tiny targets, or cluttered regions, call for executable short-horizon operations, while more complex tasks also depend on high-level workflow priors for tool orchestration and multi-step reasoning. Therefore, we hierarchically organize the skill space into two levels: an execution-skill layer \(\mathcal{K}\) and a workflow-skill layer \(\mathcal{H}\).

3.2.1 Execution Skills.↩︎

Execution skills encode reusable operations for resolving local visual bottlenecks. Each execution skill is represented as a tuple \(k=(c,u,e)\), where \(c\) denotes a textual trigger condition specifying when the skill applies, \(u\) denotes the tool action or action template to be used, and \(e\) denotes the expected local effect. This representation makes each skill explicit about when it applies, how it should be used, and what local evidence improvement it is expected to produce. To distill execution skills, we employ a specialized MLLM named \(\mathrm{\small MLLM}_{\mathrm{dk}}\) for skill-library operations. Given a successful trajectory \(\tau\) and its associated task input \((q,I)\), we prompt \(\mathrm{\small MLLM}_{\mathrm{dk}}\) to summarize reusable local tool-use patterns into execution-skill tuples: \[\{k_j\}_{j=1}^{K_\tau}=\mathrm{\small MLLM}_{\mathrm{dk}}(q,I,\tau),\] where \(K_\tau\) is the number of execution skills distilled from \(\tau\). Each \(k_j\) is a compact textual abstraction of a reusable local pattern, rather than a raw trajectory fragment. For example, a successful trajectory may reveal that rotated evidence can be resolved by image rotation, while small or boundary-adjacent evidence can be addressed by local cropping. Because execution skills are represented as condition–action–effect tuples, they can be directly matched and reused based on visual conditions, without requiring similarity to an entire trajectory.

3.2.2 Workflow Skills.↩︎

We introduce workflow skills to capture workflow-level priors for multi-step tool orchestration and reasoning. Each workflow skill is represented as \(h=(b,w),\) where \(b\) denotes a coarse-grained summary of the visual bottleneck (e.g., cross-region consistency checking), and \(w\) denotes a workflow sketch that summarizes the corresponding high-level solution pattern (e.g., locate evidence \(\to\) enhance local content \(\to\) verify consistency). Unlike execution skills, workflow skills do not directly encode executable actions. Instead, they distill workflow-level patterns for orchestrating tool use from successful trajectories. Consitent with the execution skills, workflow skills are distilled with \(\mathrm{\small MLLM}_{\mathrm{dk}}\) from the task query \(q\), the image input \(I\), and a successful trajectory \(\tau\): \[h=(b,w)=\mathrm{\small MLLM}_{\mathrm{dk}}(q,I,\tau),\] where \(b\) denotes the bottleneck description and \(w\) denotes the corresponding workflow sketch. This distillation step converts successful trajectories into compact workflow-level abstractions that are transferable across tasks and suitable for retrieval.

3.2.3 Skill Library Consolidation.↩︎

Without consolidation, newly distilled skills would be inserted into the libraries directly, causing two issues: increased retrieval noise from near-duplicate entries and uncontrolled library growth due to entry proliferation. To address this, we apply a unified merge-or-add rule to both libraries. For each newly distilled skill candidate, the system checks whether any existing entry is sufficiently similar in the embedding space. For execution skills, similarity is computed based on the trigger condition. Given a candidate \(\hat{k}=(\hat{c},\hat{u},\hat{e})\), the system checks whether any existing execution skill \(k=(c,u,e)\in\mathcal{K}\) has cosine similarity above a threshold \(\theta_{\mathrm{sim}}^{K}\) with \(\hat{c}\). If such an entry exists, the candidate is merged into the most similar matched skill by refining the items; otherwise, it is added as a new entry. For workflow skills, matching is performed using the bottleneck description. Given a candidate \(\hat{h}=(\hat{b},\hat{w})\), the system checks whether any existing workflow skill \(h=(b,w)\in\mathcal{H}\) has cosine similarity above \(\theta_{\mathrm{sim}}^{H}\) with \(\hat{b}\). If so, the candidate is merged into the matched entry; otherwise, it is inserted as a new workflow skill. To further control library size, we record the number of similarity-based matches for each entry and prune the least frequently matched ones whenever the library exceeds its capacity.

3.3 Skill and Policy Co-Evolution↩︎

During this stage, policy optimization is performed on skill-conditioned rollouts at every training step, whereas the skill libraries evolve periodically every \(N\) steps. This design enables a closed loop of co-evolution between the policy and the skill library: retrieved skills provide structured guidance for policy improvement, and the improved policy in turn produces higher-quality rollouts from which better skills can be distilled.

3.3.1 Hierarchical Skill Retrieval.↩︎

We adopt a two-stage retrieval procedure aligned with the hierarchy of workflow skills and execution skills. The key idea is to first retrieve high-level workflow priors, and then retrieve task-relevant execution skills conditioned on the identified bottleneck. Given a task \(T=(q,I)\), the actor model first produces a coarse bottleneck query \(\tilde{b}_{T}={MLLM}_(q,I).\) This query is then used to retrieve the top-\(K\) workflow skills whose bottleneck descriptions are most similar in the embedding space: \(\mathcal{H}_{T}^{\mathrm{top}} = \operatorname{TopK}_{h=(b,w)\in\mathcal{H}} \;\mathrm{sim}(\tilde{b}_{T}, b).\) The actor model then further decomposes the task into fine-grained local bottleneck descriptors conditioned on the retrieved workflow priors: \[\mathcal{B}=\{b^{(1)}, \dots, b^{(M)}\} = \mathrm{\pi_{\theta}}(q,I,\mathcal{H}_{T}^{\mathrm{top}}).\] Each descriptor \(b^{(m)}\) specifies a local visual obstacle, such as viewpoint mismatch, a small target, or dense text. For each descriptor, we retrieve the best-matching execution skill by comparing it with the trigger conditions of skills in the execution-skill library \(\mathcal{K}\): \[k^{*(m)} = \arg\max_{k=(c,u,e)\in\mathcal{K}}\; \cos\!\big(E_{L}(b^{(m)}),\, E_{L}(c)\big).\] If the best similarity score falls below a threshold \(\theta_{sim}\), the descriptor is discarded. The retained set of execution skills is \[\mathcal{K}_{T}^{L} = \left\{ k^{*(m)} \;\middle|\; \max_{k\in\mathcal{K}}\, S_{L}(k\mid b^{(m)})\ge\tau_{sim},\; b^{(m)}\in\mathcal{B} \right\}.\]

The policy then selects actions conditioned on both the retrieved workflow skills and the retrieved execution skills: \[a_t \sim \pi_{\theta}\!\left(a_t \mid o_{\le t},\, q,\, I,\, \mathcal{H}_{T}^{\mathrm{top_{1}}},\, \mathcal{K}_{T}^{L}\right),\] where \(o_{\le t}=(o_0,\dots,o_t)\) denotes the observation history. The resulting skill-conditioned trajectories are scored using the reward defined in Section 3.1 and optimized with the GRPO objective. This hierarchical retrieval guides rollout generation with reusable workflow-level and execution-level priors, rather than leaving policy updates to rely on generic task interactions alone. As a result, it improves both the efficiency and the stability of policy optimization.

3.3.2 Online Skill Evolution.↩︎

Skills induced from early, weaker rollouts may become suboptimal as the policy improves. We therefore perform periodic skill library evolution using newly collected high-quality rollouts. After every \(N\) policy steps, we open an evolution window indexed by \(e\). Rather than updating the libraries from all successful trajectories, we retain only the top trajectories ranked by GRPO relative advantage \(A(\tau)\), which preserves diverse but high-quality interaction patterns for subsequent skill evolution. For workflow skills, let \(\mathcal{T}_{e}^{\mathrm{wf}}\) denote the retained top-\(\rho_{\mathrm{wf}}\) successful rollouts in window \(e\) for task \(T\). We distill a new workflow-skill candidate with \(\mathrm{\small MLLM}_{\mathrm{dk}}\) as \[\hat{h}_{e}=\bigl(\hat{b}_{e},\,\hat{w}_{e}\bigr) = \mathrm{\small MLLM}_{\mathrm{dk}}\!\left(q,I,\mathcal{T}_{e}^{\mathrm{wf}}\right),\] where \(\hat{b}_{e}\) is the updated bottleneck description and \(\hat{w}_{e}\) is the corresponding workflow sketch. The candidate is then merged into, or added to, the workflow-skill library \(\mathcal{H}\) using the consolidation rule in Section 3.2. Execution skills are distilled in the same manner from the retained top-\(\rho_{\mathrm{ex}}\) successful rollouts for execution-skill evolution. To control library growth, we further prune the least useful entries of execution skills when the library exceeds its capacity. In particular, we rank execution skills by their empirical success rate and remove the lowest-ranked ones, where the empirical success rate of an execution skill \(k\) is defined as \(\mathrm{SR}(k)=\frac{N_{\mathrm{success}}(k)}{\max\!\left(N_{\mathrm{retrieval}}(k),1\right)}.\) This periodic evolution closes the loop: stronger policy produces higher-quality rollouts, which yield better skills, which in turn provide stronger priors for subsequent rollouts. The co-evolution mechanism is what distinguishes our method from static memory systems that only retrieve at test time.

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.1 Tasks and Data.↩︎

Following CodeVision [12], we employ a two-stage post-training recipe, with supervised fine-tuning (SFT) followed by reinforcement learning (RL). In the SFT stage, we use all publicly available CodeVision-SFT data (about 5K samples). In the RL stage, we start from the released 34,795 CodeVision-RL samples and refine them through several filtering steps, including tool-demand screening, tool-type balancing, duplicate removal, and rollout-cost control, resulting in a final set of about 12K samples. We run evaluation across three multimodal domains: (1) TIR-Bench [28] for agentic thinking with images; (2) MathVerse [29], MathVision [30], WeMath [31] and ChartQAPro [32] for multimodal reasoning; (3) V* Bench [33], HRBench-4K [34], and HRBench-8K [34] for visual search.

4.1.2 Baselines and Metrics.↩︎

We compare our method against baselines from three categories: (1) prompt-based methods, including Prompt with and without Tool Examples; (2) RL-based methods, including GRPO [13] and RLOO [27]; and (3) memory-based methods, including MemP[18], Dynamic Cheatsheet [17], and Agent-KB [19]. The details regarding baseline implementations can be found in the supplementary materials. For multimodal reasoning and visual search benchmarks, we report the Task Success Rate. For agentic thinking with images, we report Task Success Rate and Avg. Tool Calls to evaluate effectiveness and efficiency, respectively.

4.1.3 Implementation.↩︎

Our method is trained on Qwen3-VL-4B-Instruct[35] and Qwen3-VL-8B-Instruct[35] as the backbone models. For skill distillation, we employ Qwen3-VL-235B-A22B-Instruct  [35] as \(\mathrm{\small MLLM}_{\mathrm{dk}}\), and for retrieval, we use text-embedding-3-small [36]. For the SFT stage, we train for 2 epochs with a batch size of 128, a learning rate of 5e-6, a cosine learning rate scheduler, and a warmup ratio of 0.05. For the RL stage, we first train one epoch for skill library initialization, and then conduct one epoch with skill-conditioned optimization. We set the learning rate to 1e-6 and the batch size to 64, and perform 8 rollouts per sample. The coefficients for the tool-use penalty is set to 0.05. For evaluation on the benchmark, we retain only skill retrieval and remove all other components, in order to provide a clean assessment of the effectiveness of our framework. For more detailed information on training and evaluation settings, please see the supplementary material.

4.2 Main Results↩︎

4pt

The results in Table 1 and Table 2 show that our method achieves the best performance across all benchmarks. Compared with prompting alone, the gains are substantial, especially on interaction-heavy tasks such as V* and HRBench, showing that simple in-context tool demonstrations are insufficient to fully unlock effective multimodal problem solving. Our method also remains consistently stronger than vanilla RL baselines such as GRPO and RLOO, indicating that the improvement does not merely come from policy optimization, but from the structured skill guidance introduced by our framework. In addition, although memory-based methods provide moderate improvements over prompting, they still lag behind our method, suggesting that naive experience retrieval is less effective than explicitly organizing and reusing transferable skills. The category-level results on TIR-Bench provide a more detailed view of where the gains arise. Our method achieves the best overall score (All = 32.0) while using fewer tool calls than all RL-based and memory-based baselines, indicating a more favorable effectiveness–efficiency trade-off. The improvements are concentrated in categories that involve sequential transformations and nontrivial tool orchestration, including OCR, Maze, Math, IR (Instrument Reasoning), SD (Spot Difference), JG (Jigsaw Game), VS (Visual Search), and RG (Rotation Game). These tasks require the agent to not only invoke tools, but also decide when to apply them, how to chain them, and how to exploit intermediate outputs. This pattern is consistent with the design of skill-policy co-evolution: the evolving skill library provides structured guidance for policy learning, while the progressively improved policy, in turn, facilitates the extraction of more advanced skills from interaction trajectories.

4.3 Ablation Studies↩︎

We conduct ablation experiments to quantify the contribution of each component, with the results reported in Table 3. We highlight three findings. (1) Skill Structure. Both levels of the hierarchical skill structure are important. Removing high-level workflow skills reduces the average accuracy by 4.0% and increases the average number of tool calls from 4.83 to 5.91, indicating that high-level skills play a central role in global planning and in coordinating multi-step tool-use workflows. Removing low-level execution skills also leads to a clear degradation, with accuracy dropping to 28.7% and tool calls increasing to 5.33. These results show that the two skill levels are complementary: high-level skills provide coarse-grained procedural guidance, while low-level skills support accurate local execution. (2) Skill Evolution. Online skill evolution is beneficial beyond a static skill library. When the skill set is frozen, the average accuracy decreases by 3.1%, and the average number of tool calls rises from 4.83 to 5.54. This result suggests that continued skill refinement during training remains important even after an initial library has been constructed, since the improved policy can in turn extract and consolidate more effective skills from new interaction experience. (3) Reward Design. The tool-call penalty is important not only for efficiency but also for overall task performance. Removing the tool-call penalty lowers accuracy by 2.6% and substantially increases tool calls from 4.83 to 6.83, showing that the penalty helps prevent unnecessarily long tool-use chains and stabilizes learning. In contrast, always penalizing tool calls reduces the average number of calls to 3.96, but causes the largest performance drop, down to 26.1%. This indicates that overly aggressive regularization suppresses useful exploration and discourages necessary tool use. Overall, the full reward design achieves the best trade-off between task accuracy and tool-use efficiency.

4.4 Analysis↩︎

We provide a detailed analysis of the evolution dynamics for skills. For more analysis, please refer to the supplementary.

Skill Library Evolution. We monitor the number of workflow skills, the number of execution skills, and the average empirical success rate of execution skills, denoted by \(\mathrm{SR}(r)\), during the skill-policy co-evolution stage. As shown in Figure 4, during the early stage of training, the number of execution skills continues to increase over training, while their average empirical success rate rises. This suggests that policy improvement not only induces the acquisition of new execution skills, but also helps refine existing ones through merging and updating. After around 100 training steps, the library size no longer grows noticeably, as it has reached the predefined capacity limit. Skill evolution is reflected primarily in consolidation and refinement of existing entries rather than further expansion of the library. In contrast, the number of workflow skills remains relatively stable throughout training, with only modest variation over time. This reflects the different functional roles of the two skill levels. Execution skills represent local visual operations, and thus continue to expand as improved policies encounter and solve a broader range of local interaction patterns. Workflow skills, in comparison, represent abstract workflows and tool orchestration strategies, so their evolution mainly takes the form of merging and updating existing structures rather than increasing their count. These observations are consistent with the proposed dynamics of skill–policy co-evolution, in which the skill library provides increasingly effective guidance for policy learning, and the improved policy in turn induces richer and more reliable hierarchical skills from ongoing interaction experience.

Training Convergence and Tool-Use Behavior. We compare our method with GRPO in terms of both convergence behavior and tool-use dynamics. As shown in Figure 5, our method achieves substantially faster improvement in success rate on valitaion set during the early stage of training, with a clear performance gap already emerging after the first few updates. It also converges to a markedly higher success rate, whereas GRPO improves more gradually and remains at a lower, less stable plateau. The difference in tool-use behavior is equally informative. For our method, the number of tool calls increases in the early phase, indicating active exploration of potentially useful operations, but then gradually decreases as training proceeds. This suggests that skill-policy co-evolution does not simply encourage more tool use; rather, it helps the agent convert early exploration into more selective and efficient tool invocation. In contrast, GRPO exhibits a noisier trajectory: tool calls continue to fluctuate at a relatively high level even after success-rate improvement has largely slowed down. Taken together, these results suggest that our framework improves not only final performance but also training efficiency by enabling the policy to discover useful tools early and then compress them into a more disciplined decision strategy.

Case Study. Figure 3 presents an illustrative comparison between a policy without retrieved skills (only GRPO) and our method with retrieved skills. Without skills, the agent applies tools in a generic and poorly ordered manner: it first crops a local region before correcting the global orientation, and then directly infers the reading from incomplete local evidence, which leads to an incorrect answer. In contrast, with retrieved workflow and execution skills, our method first follows a high-level workflow that resolves the global ambiguity, isolates the relevant evidence, and then verifies the final reading. This workflow is instantiated by targeted execution operations including rotation correction and local crop-based inspection, enabling the agent to focus on the informative scale region and produce the correct answer more reliably.

5 Conclusion↩︎

In this paper, we propose a skill-policy co-evolution framework for multimodal agents that think with images, which jointly optimizes policies and a hierarchical skill library during reinforcement learning. By distilling successful trajectories into reusable low-level execution skills and high-level workflow skills, our method enables skills and policy to co-evolve over time. Experiments on three categories of benchmarks show that our method consistently outperforms strong baselines. Further ablations confirm that both the hierarchical skill design and the co-evolution mechanism are critical to the final gains. The analyses further highlight the notable skill evolution dynamics exhibited by our method. Overall, our results suggest that jointly evolving skills and policy provides an effective framework for building more capable and efficient multimodal agents.

References↩︎