1 Introduction↩︎

Agents are moving beyond prompt-only chat interfaces toward workflow execution [1], [2]. This shift is especially important for image generation, where prompt-only interfaces obscure many of the decisions that shape the final result, such as how conditioning is applied, how visual components are composed, and how generated image failures are detected and repaired. Workflow systems such as ComfyUI [3] expose these decisions as editable pipelines, making image generation more inspectable, controllable, and reusable [4], [5]. At the same time, this fine-grained control turns image creation into a workflow problem: an agent must select compatible components, satisfy model constraints, diagnose visual failures, and repair the workflow without breaking execution. Because similar workflow patterns recur across image-generation tasks, effective agents must do more than plan and reflect within a single run: they should also acquire reusable skills from prior executions to avoid repeating the same errors [6], [7].

This shift changes the agent’s role from specifying prompts to operating an executable procedure. Each step is mediated by a harness that exposes graph edits, runtime feedback, and recovery mechanisms; one invalid operation can break execution, and some failures are only revealed after the workflow is run [4], [8]–[15]. Recent workflow-generation agents reduce this burden by generating or refining workflows from natural language [2], [16], [17], while frameworks such as GEMS show promises of closed-loop refinement, memory, and skills for multimodal generation [18]. However, their control interfaces and skill libraries are static, or updated only by passively storing experience as memory rather than actively refining it into reusable skills. Thus, a refinement that succeeds in one run rarely becomes a validated procedure that can be invoked safely in future workflows [8].

This limitation motivates us that workflow agents need infrastructure for both controlling the current executable state and carrying useful experience for future tasks. Turning a workflow repair into reusable workflow competence requires two components: a harness that exposes tools, feedback, memory, and state transitions to the agent [19]–[24], and skill management, which converts repeated trajectories and past execution experiences into reusable procedural knowledge [8], [25]–[30]. A harness without evolving skills repeatedly rediscovers similar workflow repairs, while skills without a strong harness can become brittle instructions that cause errors for the current workflow. Thus, the challenge is not only to refine a workflow during one run, but to turn feedback from that run into reusable control knowledge for future runs. This motivates our question: can feedback from an agent’s self-verifier support both immediate workflow repair and the long-term evolution of a reusable skill library?

Inspired by prior work on agentic workflow control and self-evolving agents [4], [25], [28], [31], we present ComfyClaw, a self-evolving framework for controlling image-generation workflows in an unmodified ComfyUI runtime. We cast workflow execution as a skill-augmented Markov Decision Process over executable graph edits, runtime feedback, verifier feedback, and reusable skills, making harness design and skill reuse explicit parts of the control problem. Built on this formulation, ComfyClaw combines typed graph editing for workflow construction, VLM-based verification for diagnosing and repairing visual failures, and a skill-evolution loop that proposes, validates, and commits reusable Agent Skills. We evaluate ComfyClaw across four benchmark splits, two image backbones, and three agent models, and find that it achieves the best average score, outperforming harness-only control by 4 absolute points and no-refinement control by 10 absolute points. Human annotators further prefer ComfyClaw on 2,400 images, and 318 evolved skills account for roughly \(50\%\) of later skill invocations.

2 Related Work↩︎

Workflow graphs as controllable creative artifacts. Workflow graphs are common in creative software, from Blender nodes [32] and Houdini networks [33] to Nuke compositing [34] and Unreal Blueprints [35]. In these systems, the graph is the artifact: it exposes intermediate structure, supports reuse, and makes complex pipelines easier to inspect than monolithic code. ComfyUI [36] brings the same idea to diffusion-based generation, where users build image, video, and audio-visual pipelines from node graphs [2], [37]. Effective use, however, still depends on knowledge of node compatibility, model constraints, and scattered community recipes [5], [37]–[40]. Recent work therefore treats ComfyUI as an agent-control target: ComfyGen [4] selects workflows from prompts, ComfyGPT [5] and GenAgent [16] synthesize graphs or code through multi-agent collaboration, ComfyUI-R1 [2] studies RL-tuned reasoning, and ComfyBench [17] provides an evaluation setting. These systems mostly optimize workflow generation or refinement for a single prompt. In contrast, ComfyClaw treats workflow construction as closed-loop control: it edits executable graphs through a typed, stage-gated harness, repairs failures with localized verifier feedback, and promotes reusable skills only after held-out validation under a graph-complexity prior.

Harnessed agents with reusable skills. Recent agent systems use explicit skills, tool interfaces, and runtime harnesses to make long-horizon execution more reliable. Voyager [8] set an influential template, where an LLM agent improves through an automatic curriculum, executable skill library, and iterative self-verification. Later work developed related pieces, including reward design in Eureka [41], critique and reflection in Self-Refine [19] and Reflexion [20], desktop-task scaffolding in OS-Copilot [9], and declarative pipeline construction in DSPy [42]. A parallel line treats skills as reusable agent artifacts. The Anthropic Agent Skills specification [43] defines a lightweight SKILL.md format with progressive disclosure, used by systems such as Claude Code [44], Hermes Agent [45], and OpenClaw [46]. Recent methods also learn or revise skills from experience: SkillRL [26] builds a hierarchical SkillBank, EvoSkill [47] mines and repairs skills from failures with held-out validation, and COS-PLAY [48] co-evolves decision and skill-bank agents from unlabeled game rollouts. ComfyClaw brings this idea to workflow control: it combines typed, stage-gated graph editing with localized verifier feedback and held-out validation, so workflow skills can be learned, tested, and reused rather than manually written or statically retrieved.

Skill-centric agents and self-improving harnesses. LLM agents are increasingly designed to act on a user’s behalf over long horizons, often through harnesses that combine tool use, memory, delegation, and reusable skills. Recent open-source systems illustrate this shift: DeerFlow 2.0 [49] uses a LangGraph-based harness with sandboxed execution, persistent memory, sub-agents, and extensible skills; OpenClaw [46] builds a multi-channel assistant around skill-based operation; and Hermes Agent [45] emphasizes reusable skills and autonomous skill creation. A related research line studies skill accumulation and self-improvement as learning objectives, including XSkill [50], EvolveR [25], and broader surveys of self-evolving agents [29], [30]. Other systems target human-facing long-horizon work more directly: Odysseus [51] studies stable RL for vision-language models (VLM) agents in long-horizon visual control, while AgentLab [52] executes research workflows with human feedback. These systems show the promise of skill-centric agents, but their skills are often used in broad, open-ended settings where effectiveness and reuse are hard to evaluate.

3 Method↩︎

ComfyClaw is an agentic framework for workflow-based image generation with reusable workflow skills. It has three components: workflow construction, verifier-guided refinement, and skill evolution. Given a prompt \(p\), an LLM agent constructs a ComfyUI workflow graph through typed edits and submits it to the runtime for rendering. A VLM verifier scores the image against the prompt and returns localized repair feedback. The agent uses this feedback, along with runtime errors, to refine the workflow over multiple iterations. Across prompts, recurring successes and failures are distilled into validated Agent Skills, stored in a skill library, and retrieved in future runs. Figure 1 summarizes the pipeline.

3.1 Preliminaries: Workflows and Agent Skills↩︎

ComfyUI workflow graphs. We represent a ComfyUI workflow as a directed graph \(G=(V,E)\). Nodes \(V\) define the operations in the generation pipeline, including text encoding, latent sampling, LoRA loading, regional conditioning, upsampling, inpainting, and image decoding. Edges \(E\) carry intermediate outputs from one operation to the next. Given a prompt \(p\), ComfyUI executes the graph to render an image or video. This graph-based interface gives the agent more control than prompt-only generation, since it can revise both the prompt and the pipeline that processes it.

Workflow editing. We treat workflow construction as a sequence of graph edits. The agent can add or remove nodes, connect or disconnect edges, and adjust node parameters. Once constructed, the workflow is submitted to ComfyUI for execution.

Agent skills. Agent skills are reusable procedures stored as SKILL.md files. Each skill contains a brief description, optional triggers, and step-by-step instructions for editing a workflow graph. To keep the context small, the agent is initially shown only lightweight skill metadata, such as the skill name and description, and retrieves the full instructions only when it decides that a skill is relevant. This progressive-disclosure design lets skills serve as retrievable workflow-editing knowledge that can be reused across runs and revised after completed editing attempts.

3.2 Workflow Construction↩︎

Planning and initialization. Given a prompt \(p\), the agent first enters a planning stage, where it identifies the target image-generation model, reviews relevant tools and skills, and decides how to construct the workflow. We represent the workflow as a directed acyclic graph (DAG), \(G_t = (V_t, E_t)\), where \(V_t\) denotes the set of workflow nodes and \(E_t\) denotes the data-flow edges at refinement step \(t\). The agent begins by constructing an initial workflow \(G_0\), a minimal spine graph that loads the diffusion model, encodes the prompt, samples the latent representation, decodes the image, and saves the output. This initial graph provides a valid starting point for later refinement.

Graph editing. The agent refines the workflow through a sequence of graph-editing actions, \[G_{t+1} = \mathrm{Edit}(G_t, a_t), \qquad a_t \in \mathcal{A},\] where \(\mathcal{A}\) includes node insertion, node connection, parameter updates, prompt edits, LoRA insertion, regional conditioning, and refinement passes. The objective of workflow construction is to produce a final workflow \(G_T\) that reaches the verifier reward threshold (Section 3.3).

Skill retrieval. Skills are available throughout workflow construction but are exposed selectively through a trigger-based router. Each skill \(\sigma_i \in \Sigma\) includes lightweight metadata, such as a name, short description, tags, and trigger phrases. Given the current workflow stage and verifier feedback, the agent scores each skill by its relevance to this metadata, and only the top-\(K\) skills are shown initially. The full skill content is retrieved only when needed. This progressive exposure keeps the context small while still allowing access to detailed procedures for workflow construction and repair.

Construction stages. After creating the spine graph, the agent enters the construction stage, where it edits the workflow DAG by adding or connecting nodes, setting parameters, and modifying the graph structure to satisfy the prompt requirements. It then proceeds to an enhancement stage, where it can apply higher-level workflow changes such as adding LoRA modules, regional attention, mask-based conditioning, or refinement passes. Together, these stages correspond to the planning, construction, and enhancement blocks shown at the top of Figure 1.

3.3 Verifier-Guided Refinement↩︎

After the agent submits \(G_t\), ComfyUI renders an image \(I_t\) which is passed to a VLM verifier. The verifier first decomposes \(p\) into a checklist of observable binary requirements \(Q=\{q_i\}\). For each \(q_i\), it returns a binary pass/fail label grounded in the image, together with a short natural-language justification; in addition it returns a holistic detail score \(s_{\mathrm{det}}\!\in\![0,10]\) that captures overall fidelity, composition, and absence of visual artifacts. The harness combines the two signals into the scalar reward of Eq. 1 .

3.3.0.1 Vision-Language Model (VLM) as a Verifier.

We use the agent as verifier to evaluate whether a generated image satisfies the prompt. The verifier decomposes the prompt into observable requirements \(Q=\{q_i\}\), such as object count, attribute binding, spatial relation, style, and anatomy. It returns requirement-level pass or fail labels, a holistic detail score, localized failure descriptions, and concrete suggestions for workflow edits. The score used by the harness is \[r \;=\; 0.6\cdot\!\frac{|Q^{\mathrm{pass}}|}{|Q|} \;+\; 0.4\cdot\!\frac{s_{\mathrm{det}}}{10}, \label{eq:reward}\tag{1}\] where \(s_{\mathrm{det}}\) is an additional one-pass quality score, ranging from 1 to 10, that measures the overall image quality and its alignment with the prompt.

3.3.0.2 Workflow Refinement loop.

The verifier emits three structured components that drive the next iteration: (i) the failing-requirement set \(Q\!\setminus\!Q^{\mathrm{pass}}\), (ii) localized natural-language descriptions of what went wrong (e.g.the leftmost figure has three arms), and (iii) concrete edit suggestions phrased in workflow terms (apply regional prompting to isolate the throwing arm). These pieces are appended to the agent’s next-iteration context and also forwarded to the skill-evolution loop (§3.4). The refinement loop terminates when \(r\) exceeds a satisfaction threshold \(\tau_{\mathrm{stop}}\) or after \(K\) iterations, whichever comes first; the iteration with the highest \(r\) is committed as the final output.

3.4 Skill Evolution↩︎

The skill library serves as the long-term memory of ComfyClaw. Rather than treating each prompt independently, ComfyClaw converts recurring successes and failures across prompts into reusable workflow procedures. Skill evolution proceeds in four stages: clustering success and failure traces, proposing skill mutations, validating the proposed mutations, and committing accepted skills to the skill library.

Success and Failure Clustering. Let \(\mathcal{S}^{(m)}\) denote the skill library after evolution cycle \(m\). After a batch of \(B\) prompts, the workflow-construction loop produces traces \[\mathcal{D}^{(m)} = \left\{ \left(p_i, G_i, x_i, r_i, f_i, e_i, a_{i,1:T_i}\right) \right\}_{i=1}^{B},\] where \(p_i\) is the prompt, \(G_i\) is the final workflow, \(x_i\) is the generated image, \(r_i\) is the verifier reward, \(f_i\) is verifier feedback, \(e_i\) is a runtime error if one occurs, and \(a_{i,1:T_i}\) is the sequence of workflow actions. As shown in the bottom part of Figure 1, these success and failure traces are sent to the skill evolution module.

We divide traces into success and failure groups using the verifier reward. A trace is treated as successful if \(r_i \ge 0.9\) and as a failure otherwise. We then cluster the success and failure traces separately according to their verifier feedback, runtime errors, workflow actions, and prompt properties. Failure clusters capture recurring failure modes, such as missing objects, incorrect counting, and weak spatial binding. Success clusters capture reusable workflow strategies, such as when to add a LoRA, when to use regional attention, how to adjust guidance, or how to structure prompts.

Skill Mutation. For each cluster \(C_\ell\), we use the agent as a skill evolver to propose a mutation to the current skill library. The mutation type is selected from \[\mu_\ell \in \{\texttt{create}, \texttt{revise}, \texttt{reinforce}, \texttt{merge}, \texttt{delete}\}.\] A create mutation adds a new procedure for a pattern not covered by existing skills. A revise mutation updates the content of an existing skill. A reinforce mutation strengthens useful triggers or emphasis terms. A merge mutation combines redundant skills, and a delete mutation removes skills that are no longer useful. Applying the proposed mutation to the current library gives a candidate skill library \[\widetilde{\mathcal{S}}_{\ell}^{(m+1)} = \mu_\ell\!\left(\mathcal{S}^{(m)}\right).\]

Mutation Validation. Each candidate mutation is validated before it is committed to the skill library. For cluster \(C_\ell\), we ask the agent to synthesize three held-out prompts conditioned on the cluster identifier, the candidate skill name, and up to five example prompts from the cluster. These prompts are intended to test whether the proposed mutation generalizes beyond the examples that produced it.

Let \(\mathcal{H}_\ell\) denote the validation prompt set for cluster \(C_\ell\). We compare the average verifier reward obtained with the current skill library \(\mathcal{S}^{(m)}\) against the reward obtained with the candidate library \(\widetilde{\mathcal{S}}_{\ell}^{(m+1)}\): \[\Delta_\ell = \frac{1}{|\mathcal{H}_\ell|} \sum_{p \in \mathcal{H}_\ell} \left[ r\!\left(p;\widetilde{\mathcal{S}}_{\ell}^{(m+1)}\right) - r\!\left(p;\mathcal{S}^{(m)}\right) \right].\] The mutation is accepted only if it does not degrade validation performance \(\Delta_\ell \ge 0.\) Rejected mutations are rolled back, while accepted mutations are committed as new skill versions.

4 Experiments↩︎

In this section, we evaluate ComfyClaw on four text-to-image benchmark splits against competitive baselines. We further present ablation studies and qualitative analyses that illustrate how ComfyClaw improves through iterative workflow refinement.

Benchmarks. We evaluate on four text-to-image benchmark splits covering compositional reasoning, dense prompt following, fine-grained fidelity, and cross-lingual generalization. GenEval 2 [53] includes 800 English prompts over objects, attributes, and spatial or numerical relations, scored with the atom-level Soft-TIFA VQA judge to avoid saturation in the original GenEval. DPG-Bench [54] contains 1,065 dense English prompts with multiple objects, attributes, and relations in each sentence. OneIG-Bench [55] evaluates subject–element alignment, text rendering, reasoning content, stylization, and diversity. We use its English split (OneIG-EN, 1,120 prompts) and Chinese split (OneIG-ZH, 1,320 prompts). Together, these benchmarks contain roughly 4,300 prompts and test the dimensions where workflow evolution should differ most from prompt-only refinement.

Agents and ComfyUI models used. We use three foundation models as workflow-control agents: Claude Sonnet 4.5 [56], Qwen-3.6-35B-A3B [57], and Gemma-4-E4B-it [58]. All are natively multimodal and are used to instantiate the workflow, verifier, and skill-evolution agents. For image generation, we evaluate two ComfyUI backbones: z-image-turbo [59], a 6B text-to-image model, and LongCat-Image [60], a 6B bilingual Chinese–English model for image generation and editing. The agents edit the ComfyUI workflow graph, while the selected backbone renders the final image.

Workflow and tool setup. To closely simulate how an external agent would control a real ComfyUI deployment, we instantiate a fresh ComfyUI server endpoint for each experimental condition. Each endpoint runs an unmodified ComfyUI server, while our agent is deployed as an external plug-in that submits jobs and retrieves execution logs through the standard interface. This design allows our framework to interact with ComfyUI without modifying the application itself, making the setup easy to maintain and naturally compatible with any future ComfyUI updates.

We provide a set of workflow-editing tools for controlling ComfyUI workflows, together with four predefined visual-quality skills by [18]. During execution, the agent does more than prompt rewriting: it can edit the workflow graph, tune node hyperparameters, and, when supported by the image model, attach and configure LoRAs. The full list of predefined tools, predefined skills, and model-specific LoRA settings is provided in Appendix 7.

Image Generation Evaluation Metrics. We follow the headline metric defined by each benchmark. For GenEval2, we report the Soft-TIFA geometric mean [61] (gm), computed over the per-prompt set of VQA questions, including yes/no and counting questions [62]. For DPG-Bench, we report the Soft-TIFA arithmetic mean (am), computed over each prompt’s corresponding set of evaluation questions. For OneIG-EN and OneIG-ZH, we use VQAScore, where each image is evaluated with a single query of the form: Does this image show <prompt>? Answer Yes or No. All of these metrics rely on a VLM-as-a-judge. In our experiments, we use Qwen3-VL-8B-Instruct [63] as the primary judge model to answer the benchmark evaluation queries.

4.1 Main Results↩︎

We compare ComfyClaw against a baseline that uses only the initial tools and skill set. It does not receive verifier feedback or perform refinement; instead, it executes the workflow once and directly returns the generated image. All experiments and evaluations are run on an RTX PRO 6000 Blackwell GPU with 96 GB of memory. We report the results in Table 1. Overall, ComfyClaw outperforms Base across all four benchmarks. These results suggest two main takeaways. First, adding a VLM verifier and refinement loop helps agents construct better workflows and generate higher-quality images. Second, skill evolution further improves performance by enabling agents to summarize successes and failures, learn from past errors, and update their skill set for future workflow construction.

Evolved skills capture reusable patterns for recurring image-generation workflows. Table ¿tbl:tab:evolved-skill-examples? shows representative evolved skills from different benchmarks. Each skill is a named SKILL.md procedure that captures a recurring workflow-execution pattern rather than a one-off memory. For GenEval2, skills such as spatial-anchor-with-count, spatial-count-binding, and attribute-binding encode reusable procedures for preserving object counts, attribute bindings, and spatial layouts in compositional scenes. For DPG-Bench, skills such as material-texture-detail, precise-color-attribution, and lighting-and-reflection-detail focus on fine-grained control of material appearance, color assignment, and illumination. For OneIG-EN and OneIG-ZH, the learned skills shift toward anime-specific character and style control, including character-counting, anime-danbooru-ordering, and anime-single-character-simple. These examples show that evolved skills are not generic memories, but benchmark and style-specific reusable procedures for recurring workflow problems.

Evolved skills are heavily used by agents. We analyze the skills evolved by Claude-Sonnet-4.5 across four benchmarks and two image-generation models. As shown in Figure [fig:evolved95reads], evolved skills account for an average of \(\sim\!50\%\) of total skill reads across benchmarks (specifically \(70.0\%\) on DPG-Bench and \(56.2\%\) on GenEval2, dropping to \(16.3\%\) on OneIG-EN and \(7.5\%\) on OneIG-ZH), indicating that agents rely heavily on them when constructing image-generation workflows for dense and compositional prompts. Across the eight experimental settings, Claude-Sonnet-4.5 produced \(318\) unique skills, totaling \(4{,}768\) skill versions. These skills reflect the distinct requirements of each benchmark. GenEval2 primarily induces object-counting and attribute-binding skills; DPG-Bench emphasizes spatial relations, poses, and anatomy; OneIG-EN favors anime-style prompting and character-count control; and OneIG-ZH further introduces Chinese-specific anime conventions and resolution-tag skills.² The evolved skills provide reusable, task-specific patterns that help agents start from stronger knowledge and generate higher-quality images.

Agents perform more complex actions than simple prompt rewriting. We track the actions taken by the Claude agent when constructing workflows in ComfyClaw, as shown in Figure [fig:modification95breakdown]. Across \(35{,}612\) workflow events, prompt-text edits are the most frequent single category but account for only \(39.3\%\) of all edits. The remaining \(60.7\%\) modify other parts of the workflow, including sampler and guidance hyperparameters (\(35.8\%\)), regional or mask-based graph topology (\(11.4\%\), with a striking \(27\%\) on GenEval2), LoRA / checkpoint selection (\(7.3\%\)), and multi-pass upscaling (\(3.2\%\)). This shows that ComfyClaw improves image generation through broader workflow optimization rather than relying only on prompt refinement.

4.2 Qualitative Analysis↩︎

We further conduct a qualitative analysis of the generated images, since quantitative metrics alone may not fully capture visual quality in image generation tasks. Our goal is to compare how different harness designs affect the agent’s ability to control the workflow and produce high-quality images. We hired two annotators to rate image quality on a 1–5 Likert scale, where higher scores indicate better visual quality and stronger text-image alignment. For each benchmark and each agent, we randomly sample 50 finalized images. Annotators were shown the input prompt together with the generated images from different experimental groups and asked to assign a Likert rating to each image.³ In total, 2,400 images are annotated. We show the results in Table 2. Overall, images generated using ComfyClaw align more with input prompts and exhibit greater visual realism and aesthetics.

Closed-loop workflow refinement repairs failures that prompt-only rewriting often cannot. Figure 4 shows ComfyClaw’s refinement loop for editing the workflow. First, the edits are structural rather than purely textual prompt revision. In Figure 4 (a), the agent stacks two Z-Image LoRAs and adds a regional-attention block before the unusual attribute purple is correctly bound to all four lions. In Figure 4 (b), the agent adjusts the regional split until the spatial relation left of, the material glass, and the count three are all satisfied.

Second, refinement is not always monotonic. Some intermediate attempts are worse than the baseline, such as when the pigs disappear or the clock is lost, but the best-so-far buffer preserves the best valid candidate while still allowing the agent to learn from failed attempts and recover in later iterations.

Third, the loop is verifier-driven. Each refinement instruction directly responds to the preceding critique, such as correcting the lion color or removing an ineffective LoRA after objects disappear. Thus, the agent is not retrying blindly; it spends additional graph edits and re-renders on the specific failures localized by the verifier. These extra operations improve compositional errors, including attribute binding, object count, spatial relations, LoRA weight stacking, and hyperparameter adjustments that are difficult for single-pass prompt-only generation to fix reliably.

5 Conclusion↩︎

Agentic workflow control is a promising direction for helping users, especially beginners, operate complex tools and automate multi-step workflows. In image generation, workflow-based systems provide more controllability than prompt-only interfaces by exposing components such as model selection, conditioning, sampling, refinement, and LoRA adapters. We propose ComfyClaw, a self-evolving workflow-control framework that converts past successes and failures into reusable skills, enabling agents to improve how they construct and refine image-generation workflows over time. Our experiments show that this self-evolving skill mechanism produces higher quality and more realistic images than agent workflow control without skill evolution, and is more preferred by human annotators. These results suggest that effective visual generation agents should not only execute workflows, but also learn and evolve reusable workflow procedures from their own experience.

6 Limitations and Future Work↩︎

Our current study focuses only on image-generation workflows. However, ComfyUI also supports video-generation workflows, which require more complex node control, scene planning, frame consistency, and temporal conditioning. Video generation also requires substantially more compute and longer execution time, making agent control more difficult because long-running workflows can increase latency, timeout failures, and response errors. We therefore leave video workflow control as an important direction for future work. Future systems will need stronger workflow-control strategies, better skill management, and more efficient skill retrieval, since video workflows are typically longer and more complex than image workflows. Managing the agent context window will also become more challenging, making compact and accurate skill retrieval especially important.

7 Predefined Tools, Skills, and LoRA Settings↩︎

7.0.0.1 Workflow tools.

We expose 17 basic tools for controlling ComfyUI workflows. These tools allow the agent to inspect the current workflow, add and remove nodes, connect nodes, edit node inputs, set prompts, configure model-specific parameters, validate the workflow, and submit the final graph for rendering. Examples include add_node, set_prompt, and set_lora. Together, these tools allow the agent to construct and repair workflows at the graph level rather than only rewriting the text prompt.

7.0.0.2 Predefined skills.

In addition to evolved skills, we provide four predefined skills inspired by [18]: photorealistic, creative, high-quality, and prompt-artist. These skills provide general image-generation guidance, such as improving photorealism, increasing visual creativity, enhancing overall quality, and rewriting prompts for stronger visual composition. They serve as the initial reusable skill library before benchmark-specific skills are evolved.

7.0.0.3 Model-specific LoRA settings.

For Z-Image-Turbo, we include two LoRAs. The first is a realistic snapshot LoRA [59], [64], which encourages more photorealistic and real-life imagery. The second is an enhancer LoRA [65], which improves generation quality across a range of visual styles. LongCat-Image does not currently support LoRAs in our setup, so LoRA-based tools are disabled for that model.

2pt

8 Skill Read Statistics and Workflow Modification Breakdown↩︎

Tables 5 and 6 characterize the agent’s workflow-construction behavior from two complementary angles. Table 5 measures how much the agent relies on evolved skills versus the four predefined base skills, broken down by benchmark. Table 6 then dissects what kinds of graph edits the agent actually performs when it constructs or repairs a workflow.

Table 6 further disaggregates the agent’s edits into six semantic event categories. Prompt-text changes and sampler/guidance adjustments together account for \({\sim}75\%\) of all events across benchmarks, confirming that the agent primarily operates at the level of conditioning rather than graph topology. Structural modifications—regional masking, model swaps, and multi-pass upscaling—are more frequent on GenEval 2, which contains the most demanding multi-object layout tasks.

6pt

9 Evolved Skill Usage↩︎

Table ¿tbl:tab:top10-evolved? provides a usage-level view of the skills produced by the Claude-Sonnet-4.5. Rather than treating these counts as direct measures of skill quality, we use them to show which reusable recipes the agent actually consults when solving each benchmark. The resulting distribution separates benchmark-specific needs: GenEval 2 emphasizes spatial binding and counting, DPG-Bench emphasizes material, lighting, and scene detail, and the OneIG splits emphasize character-count and anime-style control.

3pt

We omit the run-level aggregator skills learned-successes and learned-errors, which summarize each cycle’s success and failure clusters rather than encoding topical recipes. Their raw read counts are high by construction: 3135/649/1096/226 and 379/—/53/49 for GenEval 2, DPG-Bench, OneIG-EN, and OneIG-ZH, respectively.

10 Qualitative Annotations↩︎

For the user study, each participant was given the instruction template in Box [box:eval-instructions] and asked to annotate generated images. The images were randomly shuffled so that annotators were blind to the corresponding method groups. Annotators were paid \(\$0.10\) per image, for a total of 2,400 annotations and \(\$240\) in compensation. The annotation task involved no anticipated risk to participants, and our protocol was reviewed and exempted by the Institutional Review Board (IRB).

11 LLM Prompts↩︎

This section lists the exact prompts used in the implementations of the agent loop (§3.2) and the region-level VLM verifier (§3.3). All prompts are drawn verbatim from comfyclaw/agent.py, comfyclaw/verifier.py, and comfyclaw/harness.py.

11.1 Agent System Prompt (§3.2)↩︎

The following string (_SYSTEM_PROMPT_BASE) is prepended to every agent conversation as the system role message. When a model is pinned the Pinned image model paragraph is appended. The <available_skills> XML block (skill names and one-line descriptions) is injected between the base prompt and the pinned-model paragraph.

You are ComfyClaw, an expert ComfyUI workflow engineer.  Your job is to BUILD
and GROW ComfyUI workflow topologies -- constructing complete pipelines from
scratch when the workflow is empty, and evolving existing ones in response to
the verifier's region-level feedback.

Iteration strategy
------------------
1. Call report_evolution_strategy first: state your plan and the top issue.
2. Call inspect_workflow to see the current topology.
3. **If the workflow is empty** (no nodes):
   a. Call read_skill("workflow-builder") to load architecture recipes.
   b. Call query_available_models("checkpoints") and query_available_models("diffusion_models")
      to discover available models -- NEVER guess filenames.
   c. Match the model filename to an architecture (SD 1.5, SDXL, Flux, Qwen, etc.)
      using the patterns in the workflow-builder skill.
   d. Build the full pipeline node-by-node using add_node, following the matching recipe.
   e. Use ONLY exact filenames from query results.
   f. Set detailed prompts on the CLIPTextEncode nodes.
   g. Call validate_workflow to catch wiring errors before submitting.
   h. Call finalize_workflow (it auto-validates and blocks if errors remain).
4. **If the workflow already has nodes**, follow the evolution strategy:
   a. Call set_prompt -- craft a detailed, professional positive prompt AND a strong
      negative prompt based on the user's goal (see "Prompt engineering" below).
      Do this EVERY iteration, even if you also plan structural changes.
   b. If a relevant skill is listed in <available_skills>, call read_skill to load
      its full instructions BEFORE applying that upgrade.
   c. Call query_available_models BEFORE adding any LoRA node.
   d. Apply structural upgrades (LoRA / regional / hires / inpaint).
   e. Tune sampler parameters (steps, CFG, seed) as needed.
   f. Call validate_workflow to catch wiring errors.
   g. Call finalize_workflow when done (it auto-validates).

Prompt engineering (step 3)
----------------------------
The workflow's positive prompt is pre-seeded with the user's raw goal text.
You MUST replace it with a professional-quality prompt every iteration.

Positive prompt -- structure:
  [subject & scene], [style], [lighting], [camera/lens], [quality boosters]
  * Expand every meaningful concept: vague nouns -> vivid adjectives + nouns.
  * Add artistic / photographic style: "cinematic", "concept art", "photorealistic",
    "watercolor painting", "isometric", etc.
  * Add lighting: "golden hour", "dramatic rim lighting", "neon glow", "soft diffuse".
  * Add quality boosters: "8k", "ultra detailed", "sharp focus", "ray tracing", "award winning".
  * If the image has multiple subjects/regions, describe each clearly.

Negative prompt -- always include these baseline entries, then add scene-specific ones:
  "blurry, out of focus, low quality, low resolution, noisy, grainy, jpeg artifacts,
   watermark, text, signature, ugly, bad anatomy, deformed, disfigured,
   poorly drawn hands, extra fingers, mutated limbs, cloned face, plastic skin"

Example (input: "a cyberpunk city at night"):
  positive: "a futuristic cyberpunk city skyline at night, towering neon-lit skyscrapers,
   wet reflective streets, holographic advertisements, dense rain, cinematic composition,
   dramatic volumetric lighting, wide angle lens 24mm, 8k, photorealistic, ultra detailed,
   sharp focus, ray tracing, blade runner aesthetic"
  negative: "blurry, low quality, noisy, watermark, text, bad anatomy, deformed, ugly,
   cartoon, anime, daytime, sunny, empty street"

Structural upgrade priority (iteration 2+)
------------------------------------------
When the workflow already has nodes AND verifier feedback is present:
  * Do NOT just refine the prompt -- prompt-only changes plateau quickly.
  * PREFER structural upgrades: LoRA, hires-fix, regional, inpaint.
  * If ANY region_issue has fix_strategies containing "inject_lora_*",
    you MUST attempt that structural upgrade, not fall back to prompt tweaking.
  * Combine: always refine the prompt AND add a structural upgrade together.
  * Only fall back to prompt-only when no LoRA / inpaint models are installed
    or the fix strategies are exclusively prompt-related.

Human-in-the-loop feedback
--------------------------
When the verifier feedback section starts with "## Human Reviewer Feedback",
the feedback comes from a human reviewer, not an automated VLM.
Human feedback expresses subjective preferences -- style, mood, composition,
color palette, artistic direction.  Prioritize these over structural/technical
changes.  Items prefixed with [HUMAN] are direct human requests -- address
each one specifically.  Do not second-guess or override human preferences.

[Pinned image model paragraph -- appended when a model is locked]
The image-generation model for this session is LOCKED to: {model_name}
  * DO NOT change the ckpt_name / unet_name of any loader node.
  * You MAY add LoRA loaders on top of the pinned model.
  * If the server has no models available (offline / dry-run), skip any action
    that requires model discovery and focus on prompt / sampler tuning.

11.2 Agent User Message Template (§3.2)↩︎

Each agent invocation receives a dynamically assembled user message (built by _build_user_message in agent.py). The template below shows all sections that may appear; sections are joined with double newlines and omitted when their data are absent.

## Image Goal (user's original request)
{original_prompt}

## Iteration
{iteration}

## Current Positive Prompt (baseline -- needs refinement)
{current_positive_prompt_from_workflow}
Use `set_prompt` to replace this with a detailed, high-quality version.

## Active Model
`{checkpoint_or_unet_filename}` ({architecture description from arch.yaml})

## Pre-loaded Skill: {model_skill_name}          [when a model-specific skill exists]
{full skill body -- agent does NOT need to call read_skill for this}

## Learned Skills (from past experience on this model)  [when evolved skills exist]
{description of evolved skill}  ->  read_skill("{evolved_skill_name}")

## Suggested Skills                               [when keyword-matched built-in skills exist]
These skills may be relevant: {comma-separated skill names}
Call read_skill(<name>) to load full instructions before applying.

## Verifier Feedback (previous iteration)         [from iteration 2 onward]
{VerifierResult.format_feedback() output}

REQUIRED structural upgrades (from verifier fix_strategies):
  * `inject_lora_detail`  ->  read_skill("lora-enhancement") then add_lora_loader
  * `add_hires_fix`       ->  read_skill("hires-fix") then add_hires_fix
  ...

## Human Reviewer Feedback (previous iteration)   [hybrid / human verifier mode]
{human feedback text with [HUMAN] prefixed items}
Prioritize their subjective preferences (style, mood, composition, color).

## Memory / Past Attempts
Attempt {n} (score={s:.2f}):
  Passed: {passed requirements}
  Failed: {failed requirements}
  Experience: {one-line lesson from that attempt}

## CRITICAL -- Empty Workflow -- You MUST Build From Scratch   [when workflow is empty]
The workflow is COMPLETELY EMPTY.  You CANNOT finalize without adding nodes.
Step 1: Call `read_skill("workflow-builder")` ...
Step 2: Call `query_available_models('checkpoints')` ...
...

Begin with report_evolution_strategy, then inspect_workflow,
apply your changes, then finalize_workflow.

11.3 Workflow Repair Prompt (§3.2)↩︎

When ComfyUI rejects a workflow submission (HTTP 4xx, validation failure, or execution error), the harness re-invokes the agent with the following feedback string (_build_repair_feedback in harness.py) as the verifier_feedback argument.

## ComfyUI Rejected the Workflow -- Repair Required

Your last workflow submission was rejected with the following error:
```
{verbatim ComfyUI error string}
```

**Repair protocol (follow in order):**
1. Call `inspect_workflow` to see the FULL current topology and all connections.
2. Call `validate_workflow` to get a list of graph errors (dangling refs, wrong slots).
3. For each error:
   - If a node references a nonexistent source -> fix with `connect_nodes` or `delete_node`
   - If a slot index is wrong -> `delete_node` the broken node and `add_node` a new one
     with correct wiring
   - If a model/filename is wrong -> use `query_available_models` to get exact names,
     then `set_param`
   - If a node class doesn't exist -> `delete_node` it and use a different class_type
4. Call `validate_workflow` again to confirm all issues are resolved.
5. Call `finalize_workflow` (it will auto-validate and block if still broken).

**IMPORTANT:** Do NOT just add new nodes on top of broken ones -- `delete_node` the
broken node first, then `add_node` a replacement with correct connections.

**Output slot reference:**
  CheckpointLoaderSimple -> slot 0: MODEL, slot 1: CLIP, slot 2: VAE
  UNETLoader / CLIPLoader / VAELoader -> slot 0 only
  KSampler -> slot 0: LATENT
  VAEDecode -> slot 0: IMAGE
  CLIPTextEncode -> slot 0: CONDITIONING

[When a previous VerifierResult is available:]
-- Previous Verifier Feedback (for context) --
{VerifierResult.format_feedback() output}

11.4 Verifier Prompts (§3.3)↩︎

The verifier (comfyclaw/verifier.py) uses three distinct prompts corresponding to the two-pass pipeline described in §3.3.

11.4.0.1 Pass 1 – Requirement decomposition (_DECOMPOSE_PROMPT).

A text-only call (no image) that breaks the user prompt into a list of yes/no questions. In the default batched mode (batch_mode=True) this step runs once per unique prompt and is cached for all subsequent iterations, so each unique prompt pays the decomposition cost only once.

Analyze the following image generation prompt and break it down into specific,
observable visual requirements. For each, write a yes/no question answerable from the image.

Respond ONLY with a JSON array of question strings.

Prompt: {prompt}

11.4.0.2 Pass 2 – Batched unified verification (_UNIFIED_VERIFY_PROMPT).

The default path. A single vision call answers all decomposed yes/no questions and simultaneously produces region-level issues, evolution suggestions, and a holistic 1–10 score. The image is uploaded exactly once per verify call.

You are an expert image quality analyst and ComfyUI workflow engineer.

You are given one generated image and the intended prompt. Do BOTH tasks in a
SINGLE pass and return ONE JSON object. Do not emit any prose, markdown, or
code fences -- just the JSON.

TASK 1 -- Requirement checks: Answer each yes/no question below based on the
image. Use strict "yes" / "no" answers (lower-case). Every question MUST be
answered; never skip.

Questions (answer all):
{questions_block}

TASK 2 -- Holistic analysis: Produce a short overall assessment, an integer
score (1-10), a list of region-level issues, and a list of concrete workflow
evolution suggestions.

Score rubric (integer 1-10):
  1-2: Completely wrong -- unrecognizable, no relation to prompt
  3-4: Major failures -- wrong subject, severe artifacts, missing key elements
  5-6: Partial match -- right subject but significant quality or accuracy issues
  7-8: Good -- matches prompt well with minor issues (slight artifacts, soft details)
  9-10: Excellent -- faithful to prompt, high quality, minimal or no issues

Fix strategy vocabulary (use these exact strings):
  inject_lora_detail   | inject_lora_style    | inject_lora_anatomy  | inject_lora_lighting
  add_regional_prompt  | add_hires_fix        | add_inpaint_pass     | add_ip_adapter
  refine_positive_prompt | refine_negative_prompt | increase_steps | adjust_cfg | adjust_sampler

Return exactly this JSON schema:
{
  "requirements": [
    {"question": "<verbatim question text>", "answer": "yes" or "no"}
  ],
  "overall_assessment": "<1-2 sentence overall quality summary>",
  "score": <integer 1-10>,
  "region_issues": [
    {
      "region": "<foreground subject | background | face | hands | sky | ...>",
      "issue_type": "<anatomy | texture | lighting | artifact | composition | detail | color | proportion>",
      "description": "<specific problem description>",
      "severity": "<low | medium | high>",
      "fix_strategies": ["<workflow action 1>", "<workflow action 2>"]
    }
  ],
  "evolution_suggestions": [
    "<concrete workflow change 1>",
    "<concrete workflow change 2>"
  ]
}

CRITICAL: requirements MUST contain exactly {n_questions} entries, in the same
order as the questions above.

Intended prompt: {prompt}

11.4.0.3 Pass 2 – Legacy detailed analysis (_DETAILED_ANALYSIS_PROMPT).

Used as a fallback when the unified call fails to parse, or when batch_mode=False. This call receives the image and returns the holistic analysis only; requirement checks are handled by separate per-question calls in that mode.

You are an expert image quality analyst and ComfyUI workflow engineer.

Analyze this generated image against the intended prompt, then return a JSON object with:
{
  "overall_assessment": "<1-2 sentence overall quality summary>",
  "score": <integer 1-10>,
  "region_issues": [
    {
      "region": "<specific area: foreground subject | background | face | hands | sky | ...>",
      "issue_type": "<anatomy | texture | lighting | artifact | composition | detail | color | proportion>",
      "description": "<specific problem description>",
      "severity": "<low | medium | high>",
      "fix_strategies": ["<workflow action 1>", "<workflow action 2>"]
    }
  ],
  "evolution_suggestions": [
    "<concrete workflow change 1: what to add/modify and why>",
    "<concrete workflow change 2>"
  ]
}

Score rubric (integer 1-10):
  1-2: Completely wrong -- unrecognizable, no relation to prompt
  3-4: Major failures -- wrong subject, severe artifacts, missing key elements
  5-6: Partial match -- right subject but significant quality or accuracy issues
  7-8: Good -- matches prompt well with minor issues (slight artifacts, soft details)
  9-10: Excellent -- faithful to prompt, high quality, minimal or no issues

Fix strategy vocabulary (use these exact strings):
  inject_lora_detail   | inject_lora_style    | inject_lora_anatomy  | inject_lora_lighting
  add_regional_prompt  | add_hires_fix        | add_inpaint_pass     | add_ip_adapter
  refine_positive_prompt | refine_negative_prompt | increase_steps | adjust_cfg | adjust_sampler

Intended prompt: {prompt}

References↩︎