1 Introduction↩︎

Code generation for structured data and multimodal content asks a model to write a program whose value lies in the artifact it renders: scientific figures as TikZ, charts as matplotlib, icons as SVG, mechanical parts as CAD programs, 3D scenes as Blender scripts, web applications as React projects, and circuits as Verilog RTL [1]–[11]. The task is broadly useful and hard in a specific way: the program must survive an external toolchain (a compiler, renderer, or simulator) that the model cannot observe while writing, and the artifact must then match the target in structure, semantics, and geometry. Even strong LLMs [12]–[15] routinely emit programs that do not compile, do not run, or render something far from the request [16]–[18].

Existing remedies fall into two camps. One scales up collaboration: team style multi agent frameworks [19]–[22] and role specialized pipelines [23]–[26] improve correctness but pay up to an order of magnitude token overhead, and recent agentic creation systems show the same closed loop instinct in single domains [27], [28]. The other camp keeps one model and adds self correction from runtime feedback [29]–[32], which is cheap but inherits the blind spots of a single perspective. Neither camp offers a domain general mechanism that anchors an independent reviewer to the toolchain, which is exactly where code driven creation fails.

We argue that this task needs the smallest collaborative structure in which an independent reviewer is grounded in verification, and human software engineering already has it: pair programming [33]–[35]. We propose PairCoder, a two agent framework whose core idea is to turn code driven generation into a toolchain verified dialogue: one agent writes the program, the other reviews it against concrete verification evidence, and control of the keyboard follows the diagnosis.

PairCoder instantiates this idea with three components. (1) Driver and Navigator roles with Self-Mirror prompting: the Driver generates and revises while the Navigator reviews, each under a role specific system prompt over the shared task and memory, preserving two genuinely different perspectives on the same code. (2) Verification grounded review: each candidate is compiled, executed, or rendered by the benchmark’s own toolchain, and the evidence, including diagnostics, test outcomes, and a rendering of the current artifact beside the target, is attached to the review; the Navigator must cite a concrete error or accept with [NOERROR], which prevents churn on already correct programs. (3) Error triggered role switching with a quality fallback: persistent errors swap the roles so the agent that diagnosed the fault repairs it, and an exhausted budget returns the candidate with the best verified quality rather than the last one.

Across 17 public benchmarks spanning program synthesis, multilingual code, data science, web, hardware, and six artifact modalities, under seven models from three vendors, PairCoder improves essentially every benchmark whose artifact is verifiable (Figure [fig:placeholder]), on full official metric suites rather than execution alone: Blender scene executability rises from 0.20 to 0.78, TikZ compile rate gains 10 to 30 points on every model, and chart, SVG, and CAD benchmarks improve in execution rate, SSIM, and CLIP, while on classical synthesis PairCoder stays competitive and often improves further (e.g.LiveCodeBench pass@1 from 0.94 to 0.99 at gpt-5.4-mini) at 2.9 to 9.2 times the single pass cost. A consistent pattern emerges: gains concentrate where the toolchain gives an informative oracle and the baseline leaves headroom, with ties at saturated oracles and mild regressions where the oracle is weak.

Our contributions are threefold: (1) PairCoder, a two agent framework that grounds pair programming in toolchain verification, unifying verification grounded review and error triggered role switching into one protocol across modalities; (2) the first systematic evidence that pair programming transfers from Python synthesis to structured data and multimodal artifact generation, improving executability, visual similarity (SSIM, CLIP, DINO, SigLIP-2), and geometry (Chamfer) across 17 benchmarks, seven models, and three vendors; and (3) a qualitative heuristic that gains scale with oracle informativeness and baseline headroom, characterizing when the collaboration helps, ties, or is wasted.

2 Related Work↩︎

2.1 Large Language Models↩︎

LLMs [13], [14], [36]–[40] have made code one of their most impactful application domains [41]–[43], supported by code specialized model families [15], [44]–[47] and increasingly demanding benchmarks [9], [48]–[53]. Foundation models also reach into multimodal perception, multimodal generation, and robustness [54]–[58], yet single pass inference remains prone to logical errors and hallucinations [17], [18], [59]. PairCoder is model agnostic: any of these models can play Driver or Navigator, and the collaboration adds reliability on top of whatever base capability the chosen model provides.

2.2 Structured Data Generation↩︎

A fast growing line of work, recently surveyed under the banner of multimodal code intelligence, treats programs as the medium for generating structured and multimodal content, where the value of the code lies in the artifact it renders: scientific figures [1], [60], charts and plots [2]–[4], [61], [62], vector graphics and animation [5], [63]–[65], garment patterns [66], parametric CAD [6], [67], procedural 3D scenes [7], [68], [69], front end code [8], [70], and hardware RTL [10], [11]. These benchmarks score the rendered artifact with perceptual and geometric metrics [71]–[74], yet the methods they evaluate are almost exclusively single pass generators. A parallel line of work synthesizes multimodal content directly with learned models rather than programs, spanning 3D editing, texture, rigging, skeleton, and human centric video generation [75]–[80]; PairCoder instead keeps the program as the medium, which makes the toolchain itself available as a verifier. PairCoder targets exactly this family of tasks and improves the full official metric suites rather than execution success alone; it is a concrete instance of the verification centered multimodal code intelligence that recent surveys call for.

2.3 Multi-Agent Collaboration↩︎

Two strands of work attack the brittleness of single pass generation. Multi agent systems divide labor across roles, from simulated teams and conversation scaffolds [19]–[22] to role specialized pipelines [23]–[26] and domain specific agentic creation [27], [28]; the other closes the loop with external signals, through self correction and tool grounded critique [29]–[32], learned verifiers and repair loops [81], [82], and agentic software systems [83], [84]. Team scale systems pay an order of magnitude token overhead and the domain built ones are engineered per task, while single model self correction inherits its own blind spots. PairCoder occupies the intersection they leave open: the smallest collaborative unit, two agents in the Driver and Navigator pattern of human pair programming [33]–[35], [85], with toolchain grounded review and error triggered role switching, applied as one uniform protocol from program synthesis to code driven multimodal generation.

3 Methods↩︎

3.1 Overview↩︎

The PairCoder framework, illustrated in Fig. 1, adapts pair programming to automated code generation with Large Language Models (LLMs). Two agents alternate between proposal and review. At each iteration, the Driver produces a candidate solution and the Navigator critiques it, after which the Driver revises the code. This interaction continues until the solution is accepted or the iteration budget is exhausted.

Formally, given a programming task \(q \in \mathcal{Q}\), PairCoder seeks to generate code \(c^* \in \mathcal{C}\) that maximizes both correctness and quality: \[c^* = \arg\max_{c \in \mathcal{C}} P(c \mid q) \cdot \text{Quality}(c, q)\]

3.2 Role Assignment↩︎

At each time step, a role function \(\rho: \mathcal{A} \times T \rightarrow \mathcal{R}\) assigns Driver and Navigator to the two agents, where \(\mathcal{A} = \{a_1, a_2\}\) denotes the agents, \(T\) the time steps, and \(\mathcal{R} = \{Driver, Navigator\}\) the available roles.

The Driver is responsible for code construction and revision, whereas the Navigator checks correctness, identifies deficiencies, and returns actionable feedback. A REVISE verdict is itself an error signal, and the switch is taken before the repair so that the agent that diagnosed the fault takes the keyboard and a different agent reviews the resulting fix. The default policy switches on each error (\(\eta=1\)); we also study switching only after \(\eta\) consecutive errors, a fixed interval every \(k\) rounds, and never switching (Sec. 4.6).

3.2.0.1 Driver Agent.

The Driver generates code conditioned on the task specification and interaction history. As illustrated in Fig. 2, its generation process follows: \[\label{eq:1} C_t = F_{driver}(I_{dri}, Q, M_{t-1})\tag{1}\] where \(I_{dri}\) denotes the driver instructions, \(Q\) is the task description, \(C_t\) is the candidate code at iteration \(t\), \(R_i\) is the review outcome at iteration \(i\), and \(M_{t-1}\) is the accumulated interaction history: \[\label{eq:2} M_t = \{(C_i, R_i) \mid 1 \le i \le t\}\tag{2}\]

Concretely, on the first round the Driver is prompted with the task \(Q\) alone and emits \(C_1\); on every subsequent round it receives the previous candidate \(C_{t-1}\), the Navigator’s review \(R_{t-1}\), and the same verification evidence \(\psi_{t-1}\) the Navigator saw, and is asked to return the full revised program. There is no auxiliary objective or fine tuning: the Driver is a frozen LLM steered entirely through its prompt and the role-specific system message.

3.2.0.2 Navigator Agent

The Navigator evaluates the current candidate against concrete verification evidence and returns structured feedback: \[\label{eq:3} R_t = F_{navigator}(I_{nav}, Q, C_t, M_{t-1}, \psi_t)\tag{3}\] where \(I_{nav}\) specifies navigator-specific review guidelines and \(\psi_t\) is the verification evidence attached to \(C_t\) (defined below).

The Navigator re-reads the requirements, traces the candidate on concrete inputs, checks that names and interfaces match the specification, and weighs the evidence \(\psi_t\), then issues one of two verdicts: \[R_t = \begin{cases} \texttt{[NOERROR]} & \text{accept} \\ \text{REVISE}(\Delta_t) & \text{otherwise} \end{cases}\] where \(\Delta_t\) quotes the offending line, the requirement it violates, and the specific fix. The protocol is deliberately conservative: when \(\psi_t\) already passes and the Navigator cannot point to a definite fault it must accept with [NOERROR] rather than churn working code on style or vague suspicion. The interaction history is then updated as: \[\label{eq:4} M_t = M_{t-1} \cup \{(C_t, R_t, \psi_t)\}\tag{4}\]

3.2.0.3 Verification Evidence \(\psi_t\).

Before each review the candidate \(C_t\) is run through the benchmark’s own toolchain and the result is attached to the Navigator’s prompt as evidence \(\psi_t\). In our implementation \(\psi_t\) combines up to three signals. (1) A compile or execute predicate from the renderer, interpreter, or simulator (LaTeX, CadQuery, matplotlib, the SVG rasterizer, Blender, or an RTL simulator), together with the captured diagnostic on failure. (2) For tasks that ship a specification, the outcome of a test the Navigator authors before it reviews, a test-driven check of \(\mathrm{satisfy}(C_t,\mathrm{spec}(Q))\). (3) For image conditioned tasks, a rendering of the current artifact placed beside the target, so the Navigator can cite concrete visual differences. The Driver receives the same \(\psi_t\) when it revises. This grounding is what makes the second perspective informative rather than a paraphrase of the first, and it is the only domain specific part of the loop; everything else is identical across the seventeen benchmarks.

3.3 System Components↩︎

3.3.0.1 Shared Environment.

The framework maintains a shared state \(S_t = \langle Q, M_t, \rho_t, \tau_t \rangle\) with four components. The query \(Q\) is the original task specification. The memory \(M_t\) is the running interaction history (candidates, reviews, and the attached evidence), kept per role and bounded in practice only by the model’s context window. The role assignment \(\rho_t\) records the current mapping between agents and roles, and the iteration counter \(\tau_t\) tracks collaboration progress.

3.3.0.2 Prompt construction.

Each agent is a frozen chat LLM accessed through an OpenAI compatible API and carries its own message history; the two histories together realize the shared memory \(M_t\). The role prompt is injected as a system message, a one line Driver or Navigator instruction plus an optional benchmark specific hint, and is re-asserted whenever the roles switch, which is the Self-Mirror mechanism in implementation terms. The Driver’s user message is the task \(Q\) on the first round and the previous program, the Navigator’s review, and the evidence \(\psi_{t-1}\) on every round after; the Navigator’s user message is a review instruction, the candidate program, and \(\psi_t\). For image conditioned benchmarks these messages are multimodal: the Driver is shown the target image, and the Navigator is additionally shown a rendering of the current candidate next to the target. The single model baseline is one direct generation with the same task prompt and no review.

3.3.0.3 Agent Brain.

As depicted in Fig. 3, each agent uses an LLM conditioned on a role specific prompt and the current shared state: \[\text{Brain}_{\text{role}} = \text{LLM}(P_{\text{role}}, S_t)\] where \(P_{\text{role}}\) encodes the responsibilities of the current role and \(S_t\) provides the task context, dialogue history, and current role assignment.

3.3.0.4 Action Space.

Each agent operates within a discrete action space \(\mathcal{A}_{\text{role}}\): \[\begin{align} \mathcal{A}_{driver} &= \{\mathrm{\small Generate},\;\mathrm{\small Refine},\;\mathrm{\small End}\} \\ \mathcal{A}_{navigator} &= \{\mathrm{\small Review},\;\mathrm{\small Accept}\} \end{align}\]

3.4 Collaboration Protocol↩︎

The collaboration follows an iterative refinement process formalized in Algorithm 4. The feedback loop, illustrated in Fig. 5, alternates between Navigator review and Driver revision, yielding progressive improvement over successive iterations.

3.4.0.1 Termination Protocol.

The system terminates when condition \(\Omega(S_t)\) is satisfied: \[\Omega(S_t) = \begin{cases} \text{true} & \text{if } R_t = \texttt{[NOERROR]} \\ \text{true} & \text{if } \tau \geq T \\ \text{false} & \text{otherwise} \end{cases}\] If the loop stops on [NOERROR] the accepted candidate is returned. If instead the budget \(T\) is exhausted, PairCoder does not return the last attempt but the best one seen, \(\arg\max_{C_i \in M_\tau}\mathrm{Quality}(C_i)\), where \(\mathrm{Quality}\) orders candidates first by whether the verification evidence \(\psi\) passes, then by the benchmark’s continuous score when one is available (for example structural similarity to the target), and finally by recency. This makes an unresolved review debate degrade to a safe fallback rather than a failure.

3.4.0.2 Self-Mirror Mechanism.

To preserve role consistency, each agent receives an identity aware prompt before generation or review: \[\begin{align} \text{SelfMirror}(\rho_t, Q, M_t) &= \text{concat}( \\ &\quad \text{RolePrompt}(\rho_t), Q, M_t) \end{align}\] where \(\text{RolePrompt}(\rho_t)\) explicitly states whether the agent should act as Driver or Navigator. This mechanism keeps the Driver focused on code production and revision, and keeps the Navigator focused on diagnosis, verification, and targeted feedback throughout the interaction.

3.4.0.3 Implementation.

Both arms use identical prompts and the same per benchmark extractor, which recovers the program from the raw reply only to run the verifier \(\psi\); the dialogue itself keeps the raw text. Empty or failed API responses are retried with exponential backoff and do not pollute the history. Reasoning is disabled for the main grid (a thinking-enabled study is reported separately), and the loop uses at most three to four review rounds in the main experiments (up to eight in the role-switching study of Sec. 4.6). The verifier \(\psi\) is the only per benchmark component; it wraps the benchmark’s official toolchain (compiler, interpreter, renderer, simulator, or unit tests). We log prompt and completion tokens separately for each arm, which is the basis for the token accounting in Sec. 4.5.

3.4.0.4 Role Switching Strategies.

Our framework supports heterogeneous agent configurations in which different LLMs can serve as Driver and Navigator, allowing us to leverage complementary model strengths. To optimize collaboration dynamics in this setting, we implement two switching policies \(\pi: S_t \rightarrow \{0, 1\}\):

Fixed interval switching. \[\pi_{\text{fixed}}(S_t) = (\tau \bmod k = 0)\] ensuring balanced participation every \(k\) iterations.

Error triggered switching. \[\pi_{\text{error}}(S_t) = \left(\sum_{i=\tau-w}^{\tau} \mathbb{I}[R_i = \text{REVISE}] \geq \eta\right)\] activating when error count exceeds threshold \(\eta\) within window \(w\).

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.0.1 Benchmarks.

Systematic LLM evaluation has been studied across many domains, from Chinese language understanding [52] and strategic reasoning [53] to web interface comprehension [9]. Following this tradition, we evaluate PairCoder on 17 public benchmarks organized by the artifact a program must produce. Program synthesis: LiveCodeBench [48] (recent contest problems), BigCodeBench [49] (library intensive tasks), and DS-1000 [50] (data science). Multilingual code: the C++, Java, and JavaScript splits of HumanEval-X [45]. Web applications: WebApp1K [8] (React user journeys scored by hidden Jest tests). Hardware: VerilogEval [10] and RTLLM [11] (simulation against reference testbenches). Code driven artifacts: DaTikZ [1] (caption to TikZ), Plot2Code [3], PandasPlotBench [4], and ChartMimic [2] (chart or data to matplotlib), StarVector [5] (image to SVG), GenCAD-Code [6] (image to CadQuery), 3DCodeBench [7] (text to Blender script), and P3D-Bench [86] (text to parametric 3D CAD, detailed in Sec. 4.4). Each benchmark is scored with its official metric suite: pass@1 or execution rate where unit tests or toolchains define correctness, plus SSIM [71], CLIP [72], DINO [73], SigLIP-2 [74], and Chamfer distance for rendered artifacts; full per benchmark scoring and reproducibility details are in Appendix 11.

4.1.0.2 Models and protocol.

We test seven models from three vendors: gpt-5.4-mini, gpt-5.4, and gpt-5.5 (OpenAI); doubao-1.5-lite and doubao-seed-2.0-mini (Doubao); deepseek-v3.2 and deepseek-v4-flash (DeepSeek), all with thinking disabled. The baseline is a single direct generation (one call per problem). PairCoder runs the loop of Sec. 3.2 with at most three to four review rounds: the Driver generates, the verification predicate \(\psi\) compiles, executes, or renders the candidate, the Navigator reviews the code together with this evidence (and, for visually grounded tasks, a rendering of the current artifact next to the target) and either accepts with [NOERROR] or requests a concrete revision, and roles switch when errors persist. Both arms use identical prompts and extraction; image input benchmarks are restricted to the vision capable models. We log every token in both arms.

3.2pt

4.2 Main Results↩︎

Table ¿tbl:tab:main95grid? reports the full grid: 17 benchmarks under all seven models. On the three OpenAI models PairCoder improves 44 of the 48 cells; the four exceptions are three exact ties at saturated oracles (ChartMimic execution at gpt-5.4-mini, BigCodeBench at gpt-5.5, and StarVector rendering at gpt-5.5) and one small regression (Plot2Code execution at gpt-5.5, 0.985 to 0.977, where the baseline already runs near the ceiling and SSIM still improves, see Sec. 5). Three patterns stand out. First, the largest gains concentrate where the toolchain provides the richest signal: 3DCodeBench executability rises by up to 58 points, DaTikZ compile rate by 10 to 30 points, and WebApp1K and RTLLM each by up to 22 points. Second, gains often grow with model capability rather than shrink: HumanEval-X C++ improves by 8.0, 11.0, and 31.1 points on gpt-5.4-mini, gpt-5.4, and gpt-5.5 respectively, because a stronger Navigator reviews more accurately and a stronger Driver exploits the feedback better. Third, on benchmarks whose baseline is already near the ceiling the loop terminates immediately via [NOERROR] and safely ties instead of regressing, which is the designed behavior of the conservative review protocol.

4.3 Cross-Vendor Generality↩︎

The four right columns of Table ¿tbl:tab:main95grid? repeat the study on models from two further vendors. The headline largely transfers: artifact domains with a strong toolchain signal improve on nearly every applicable model. DaTikZ gains 10 to 30 compile points on all four models, 3DCodeBench gains 22 to 33 executability points on three of four, WebApp1K gains 2.5 to 22.5 points wherever the model can write React at all, and the vision capable doubao-seed-2.0-mini improves every chart, SVG, and CAD metric. The same columns also delimit the method honestly: on benchmarks where the verification signal is weak (DS-1000 signature execution without functional tests, BigCodeBench library calls, RTLLM compile only evidence), the weaker or non GPT navigators occasionally churn working code and small regressions appear (for example DS-1000 on the DeepSeek models). This is exactly the boundary in the qualitative pattern above: where the toolchain cannot adjudicate, review quality becomes model bound; where it can, gains appear across all vendors; Appendix 9 (Figures 19–23) shows the same tasks rendered across these models side by side. As a negative control we also ran GeoCodeBench [87], a PhD level 3D vision benchmark on which both arms score at the floor (about 0.02), and PairCoder ties rather than regresses.

4.4 Generalization to Parametric 3D Code (P3D-Bench)↩︎

To test whether the verification-grounded loop transfers to a target where the artifact is a 3D solid rather than a program output, we evaluate on P3D-Bench [86], a benchmark in which the model writes a parametric construction program that is compiled to a mesh and scored against ground-truth geometry. P3D-Bench spans three tracks (text-to-3D, image-to-3D, and assembly-3D) across several program formats (minimal-JSON, OpenSCAD, CadQuery, Three.js). We evaluate the text-to-3D track primarily in the minimal-JSON format, the one configuration whose ground truth is materializable from the openly licensed Text2CAD v1.1 corpus, and additionally report executable validity for the OpenSCAD format. For the image-to-3D and assembly-3D tracks, whose full evaluation sets build on the licensed Fusion 360 Gallery, we report a three-case in-repo demonstration (Appendix 10, Table 6), on which PairCoder improves validity, geometry, and topology in the CadQuery format of both tracks and ties on Three.js, where the baseline already compiles all three cases.¹ The Navigator’s verification predicate \(\psi\) is simply the benchmark’s own compiler (“does the program build a valid solid?”), and we report the official metrics: executable validity, a geometry score (Chamfer distance, F-score, normal consistency and volumetric IoU, combined as the worst-filled composite that zeroes invalid programs), and a topology score. We run the full 400-case text-to-3D split with thinking disabled.

The P3D-Bench rows of Table ¿tbl:tab:main95grid? show the same headroom-tracking pattern as the rest of the grid: PairCoder improves every metric on every model with no regressions. The weakest baseline (doubao-1.5-lite, \(14.0\%\) valid) gains \(+28.7\) points of validity, \(+0.145\) geometry and \(+0.280\) topology, and gpt-5.4 (\(71.5\%\)) gains \(+27.5/+0.136/+0.275\), while near-saturated models such as gpt-5.4-mini (\(97.3\%\)) and deepseek-v4-flash (\(94.5\%\)) still improve rather than regress. Crucially, on programs the baseline already compiles the geometry is unchanged: PairCoder intervenes only when the artifact fails to build, so coverage rises without disturbing correct solids, the conservative-review behavior of Sec. 4.2, now measured on 3D geometry and topology. Appendix 10 breaks the geometry and topology columns into their official sub-metrics (capped Chamfer, F-score@0.05/0.01, normal consistency, and volumetric IoU for geometry, Table 3; no-open-edge, inverted-normal, and non-manifold ratios for topology, Table 4): PairCoder improves every sub-metric on every model, while on the programs the single model already compiles PairCoder keeps that program unchanged, so its geometry is identical there and the Chamfer distance on that common set matches the baseline exactly. This makes the source of the gain unambiguous: it is coverage of non-executable programs, not re-optimization of already-valid geometry: PairCoder repairs programs the single model fails to build at all into valid solids. Appendix 10 additionally illustrates a vision-grounded geometry-refinement variant that improves already-valid but inaccurate solids (Fig. 24), for example raising F-score@0.05 from \(0.23\) to \(0.95\). A second program format tells the same story: with OpenSCAD in place of minimal-JSON, PairCoder drives validity to a perfect \(1.000\) on all three models tested, though the gain is smaller because the baseline is already more fluent in OpenSCAD (Appendix 10, Table 5).

4.5 Token Economics↩︎

We log every token in both arms. Across the full grid PairCoder consumes 2.9\(\times\) to 9.2\(\times\) the tokens of the single model baseline depending on the benchmark (for example 2.9\(\times\) on GenCAD-Code, 3.4\(\times\) on StarVector, 4.0\(\times\) on WebApp1K, 5.9\(\times\) on ChartMimic, 6.9\(\times\) on DaTikZ, 7.9\(\times\) on RTLLM where simulation failures trigger the most revision rounds, and 9.2\(\times\) on Plot2Code), with an overall ratio of 7.4\(\times\) over 138M logged tokens (the per benchmark breakdown is in Table 2). This is the cost of the review rounds themselves: tasks accepted in the first round cost roughly 2\(\times\) a single pass (one generation plus one review), and the multiplier grows only when the Navigator actually finds errors, that is, exactly when the additional spend converts into accuracy (Fig. 6). Team style multi agent frameworks reported in prior work consume an order of magnitude more than a single call; while we do not re-run those systems on our benchmarks, PairCoder’s measured 3 to 9 times envelope is well below that order of magnitude overhead.

4.6 Role Switching Analysis↩︎

The error triggered role switch is a design choice; here we test it directly at gpt-5.4-mini on four benchmarks, against three alternatives at matched budget: no switch (the Driver keeps the keyboard, the Navigator always reviews), switch on each error (the Navigator that just flagged an error takes over immediately), switch after two consecutive errors, and a fixed interval switch every two rounds regardless of the signal. Figure 7 reports the official metric and the token cost for each policy.

Switching on each error is the best or tied-best policy on all four benchmarks, and never the worst. The effect is small where the loop is short (DaTikZ saturates near a perfect compile rate under every policy) but clear where iteration matters: on HumanEval-X C++, switching immediately reaches 0.427 while delaying the switch to two consecutive errors drops to 0.390, the same as never switching, because the agent that diagnosed the fault is the one best positioned to fix it, and keeping it in the reviewer seat wastes that diagnosis. No switch and the fixed interval are competitive but consistently a step behind. We therefore adopt switch on each error as the default in all other experiments. This updates the earlier preference for a two error threshold, which we had observed on Python only; on the broader benchmark set immediate switching is the more reliable choice.

4.7 Effect of Reasoning Effort↩︎

The main results disable model thinking. We additionally test whether enabling reasoning changes the picture, on the three Python benchmarks where the no-thinking gains are smallest, at gpt-5.4-mini. Enabling high reasoning effort raises both arms’ baselines, as expected, and PairCoder still helps where the verification signal is informative: on BigCodeBench the gain grows from \(+0.8\) points (no thinking, 0.450 to 0.458) to \(+5.0\) points (high, 0.425 to 0.475), because a stronger Driver produces candidates whose remaining errors are more often caught by the Navigator’s doctest evidence. On DS-1000, whose only signal is that a function executes on its signature (no functional tests), thinking lifts the baseline from 0.250 to 0.310 but PairCoder neither helps nor hurts beyond noise (0.310 to 0.300), consistent with the qualitative pattern above: where the toolchain cannot adjudicate correctness, extra reasoning improves the Driver but gives the Navigator nothing new to verify. Reasoning roughly triples per-call token cost, so we use it only for this analysis and report the main grid with thinking disabled. (We omit hardware RTL from this comparison: at high effort the small model spends most of its budget on reasoning and frequently truncates the actual Verilog module, an artifact of the model rather than of the collaboration.)

5 Code-Driven Multimodal Generation↩︎

The central claim of this work is that pair programming turns code driven creation, not just program synthesis, into a verified process. We test it on seven public benchmarks whose outputs are executable artifacts in other modalities: scientific figures (DaTikZ [1]: caption to TikZ/LaTeX), charts (Plot2Code [3], PandasPlotBench [4], ChartMimic [2]: image or data to matplotlib), vector graphics (StarVector [5]: image to SVG), parametric CAD (GenCAD-Code [6]: image to CadQuery), and 3D scenes (3DCodeBench [7]: text to Blender script). In each domain the Navigator’s review is grounded in the same verification predicates \(\psi_i\) as in Sec. 3.2: the candidate program is compiled, executed, or rendered, and the result (including the rendered image for multimodal tasks) is supplied to the Navigator as review evidence before it issues [NOERROR] or a concrete revision request. We report each benchmark’s official metric suite, namely execution or compile rate plus visual similarity (SSIM [71], CLIP [72], DINO [73], SigLIP-2 [74]) and geometric fidelity (Chamfer distance), rather than execution success alone.

3pt

5.0.0.1 Quantitative results.

Table 1 summarizes the gpt-5.4-mini results. PairCoder improves every official metric on six of the seven benchmarks: not only are far more generated programs executable, the artifacts they produce are also closer to the reference in pixel structure (SSIM), semantics (CLIP, SigLIP-2, DINO), and geometry (Chamfer). The only exception is ChartMimic, where the baseline already executes near the ceiling (about 0.97 to 1.0) and the loop ties; notably, at gpt-5.4 the same benchmark turns positive (execution 0.92 to 0.98, SSIM 0.53 to 0.58), confirming that the tie reflects a saturated oracle rather than a limit of the collaboration. These gains transfer across vendors: with the volcano engine models, DaTikZ improves on all four (doubao-1.5-lite \(+30.0\), doubao-seed-2.0-mini \(+23.3\), deepseek-v3.2 \(+23.4\), deepseek-v4-flash \(+10.0\) compile points), 3DCodeBench gains \(+21.6\)/\(+33.3\)/\(+30.0\) executability points, and the multimodal doubao-seed-2.0-mini improves nearly every chart, SVG, and CAD metric (for example, PandasPlotBench execution 0.91 to 0.99).

5.0.0.2 Qualitative comparison.

Figure 8 illustrates the dominant failure repair pattern behind these numbers. Single model generations in these domains frequently die at the toolchain boundary: LaTeX that does not compile, matplotlib scripts that raise exceptions, Blender scripts that produce no geometry. PairCoder converts a large fraction of these hard failures because the Navigator receives the concrete compiler or runtime error (and, for image conditioned tasks, a rendering of the current candidate next to the target) and returns an actionable critique; after the error triggered role switch, the agent that diagnosed the fault repairs it directly. When the baseline already renders, the same visual review instead nudges layout, proportions, and styling toward the target, which is reflected in the consistent SSIM, CLIP, and DINO gains. Appendix 8 expands this into one page per benchmark, with galleries (Figures 12–18) and the full measured metric suites (Tables [tab:quant95plot2code]–[tab:quant95tdcb]); Appendix 7 walks through complete multi-round traces (Cases 7.1–7.4).

5.0.0.3 Cost.

The collaboration costs about 3 to 9 times the tokens of single model inference in these domains (for example, 2.9\(\times\) on GenCAD-Code, 3.4\(\times\) on StarVector, 5.9\(\times\) on ChartMimic, 6.9\(\times\) on DaTikZ, and 9.2\(\times\) on Plot2Code, measured per arm), with an overall ratio of about 7\(\times\) across the full grid; review rounds terminate early via [NOERROR] on the majority of tasks that succeed immediately.

6 Conclusion↩︎

Code is increasingly how language models make things, from figures, charts, and vector graphics to CAD parts, 3D scenes, and circuits, and in this regime the decisive failures occur at the boundary between the program and the toolchain that renders it. We presented PairCoder, a minimal two agent framework that closes this boundary: a Driver writes the program, a Navigator reviews it with verdicts grounded in compiler, execution, and rendering evidence, and error triggered role switching keeps the collaboration moving. Across 17 public benchmarks, seven models, and three vendors, PairCoder improves essentially every verifiable benchmark on its entire official metric suite, covering executability, visual similarity, and geometric fidelity alike, while remaining within 2.9 to 9.2 times a single pass in cost (about 7 times overall), far below team simulation systems; on classical synthesis it remains competitive with and often improves strong single model inference (for example LiveCodeBench pass@1 from 0.94 to 0.99). The broader lesson is a qualitative pattern for collaborative generation: gains concentrate where the artifact is verifiable and the baseline leaves headroom, and fade where the oracle is weak. Wherever a trustworthy toolchain can be made to speak, and in code driven creation it almost always can, pair programming converts that signal into reliably better artifacts, making it a practical recipe for trustworthy code mediated generation of structured data and multimodal content wherever such an oracle exists.

Limitations↩︎

While PairCoder improves code generation accuracy, it still requires substantially more tokens and wall clock time than single model inference, with about seven times the cost per task on average in our experiments (2.9 to 9.2 times depending on the benchmark). Although one PairCoder run can reach accuracy levels that otherwise require several retries, the added latency and token budget still constrain practical deployment. Reducing this cost while preserving accuracy remains an important direction for future work.

7 Multi-Round PairCoder Traces on Code-Driven Multimodal Benchmarks↩︎

This appendix documents complete PairCoder sessions on the multimodal benchmarks of Sec. 5. Each trace shows the Driver’s program, the verification evidence \(\psi\) attached to the Navigator’s review (execution status and, for image conditioned tasks, the structural similarity of the current rendering to the target), the Navigator’s verbatim verdict, and the error triggered role switches. Renders of the rounds accompany each trace so the effect of each review can be inspected visually. Traces use gpt-5.4-mini unless stated otherwise; prompts and the review protocol are identical to the main experiments.

7.1 Case 1 (Plot2Code): review, role switch, and cross acceptance↩︎

The Driver reproduces a three panel errorbar chart from the target image (Fig. 9). Its first program already runs and is close to the target, but the Navigator, looking at the current rendering next to the target, finds a concrete visual deviation and rejects the round. After the role switch the former Navigator takes the keyboard and applies its own fix, and the former Driver, now reviewing, accepts with [NOERROR]. The session costs two generations and two reviews.

fig, axs = plt.subplots(1, 3,
    figsize=(12, 6), sharex=True)
fig.suptitle("Errorbar subsampling",
    fontsize=14)
axs[0].errorbar(x, y1, yerr=yerr, ...)
axs[1].errorbar(x, y1, yerr=yerr,
    errorevery=6)
axs[2].errorbar(x, y1, yerr=yerr,
    errorevery=(6, 3))

runs OK; structural similarity to
the target chart = 0.876

The code has a visual mismatch with
the target figure:
- Line causing the mismatch:
  fig.suptitle("Errorbar subsampling",
               fontsize=14)
- Why: the target image uses a larger,
  more prominent figure title and
  slightly different top spacing. With
  fontsize=14 the title is too small
  compared to the reference.
- Fix: increase the suptitle size and
  adjust its vertical placement.

- fig.suptitle("Errorbar subsampling",
-     fontsize=14)
+ fig.suptitle("Errorbar subsampling",
+     fontsize=18, y=0.98)

[NOERROR]

7.2 Case 2 (Plot2Code): review oscillation and the quality fallback↩︎

A harder session on a two panel contour task. The pair improves the program over three revision rounds (contour levels, then colorbars), but the two reviewers disagree about whether the target contains colorbars, so no round is accepted within the budget. PairCoder then falls back to the quality criterion of Algorithm 4 and returns the round with the highest verified similarity score, which is also the visually closest rendering (Fig. 10). The fallback turns an unresolved review debate into a safe outcome instead of a failure.

The code has a definite mismatch:
- Problem line:
  levels = np.linspace(-0.48, 0.40, 15)
- Why: the prompt says the plots should
  use 14 levels, and the target image's
  contour/colorbar scaling does not
  match this fixed custom range.
- Concrete fix: use 14 contour levels
  directly in both plots.

Missing colorbars.
- The target figure shows a colorbar
  beside each subplot, but the code
  never adds one. The exact missing
  lines are:
  fig.colorbar(cf, ax=ax)
  fig.colorbar(cf2, ax=ax)

The code still has a concrete mismatch:
- Problem lines: fig.colorbar(cf, ax=ax)
  and fig.colorbar(cf2, ax=ax)
- Why: the target image shows NO
  colorbars. Adding them changes the
  layout and shrinks the axes.

7.3 Case 3 (DaTikZ, doubao-1.5-lite): compile error repair on a weak model↩︎

The same protocol on the weakest model in our pool, where PairCoder lifts the DaTikZ compile rate from 0.550 to 0.850. The Driver’s first TikZ program fails to compile because its caption references an undefined cross reference macro. The Navigator receives the compiler verdict as evidence, identifies the exact offending construct, and proposes the minimal removal; after the role switch the repaired program compiles and the former Driver accepts (Fig. 11).

LaTeX compilation FAILED

The error is that the code uses
\ref{claim:paths} which is not defined
anywhere in the provided code. To fix
this, either define the claim:paths
macro appropriately in the preamble or
remove the reference from the caption.
Specific fix: remove the
\ref{claim:paths} part from the
caption.

[NOERROR]

7.4 Case 4 (DaTikZ): first round acceptance, the cheap path↩︎

For completeness, the most common trace shape: the Driver’s first program compiles, the Navigator receives the passing evidence, finds no concrete error, and accepts immediately with [NOERROR]. The session costs one generation plus one review, roughly twice a single pass, which is why the average token multiplier in Sec. 4.5 stays in the 3 to 9 range even though failing tasks iterate several times.

compiles and renders OK

[NOERROR]

8 Per-Benchmark Qualitative and Quantitative Results↩︎

This appendix devotes one full page to each artifact benchmark: a gallery of qualitative comparisons on top and that benchmark’s complete measured metric suite below. In every gallery triplet the target or ground truth rendering is on the left, the single model baseline’s artifact is in the middle, and PairCoder’s artifact is on the right (copper border), with the task id and input quoted beneath; each example is a distinct task. For the chart, plot, and vector benchmarks every example has both arms rendering with PairCoder quantitatively closer to the target, so the comparison isolates artifact quality. For DaTikZ, GenCAD-Code, and 3DCodeBench, where PairCoder’s measured advantage is concentrated in executability, the galleries lead with fidelity gains and then show executability repairs, in which the baseline produces no artifact (gray panel) and PairCoder renders successfully. Each table reports every metric we measured for that benchmark, as “single model \(\rightarrow\) PairCoder,” with green cells marking improvements and aggregate scores counting non rendering generations as zero.

5pt

9 Cross-Model Qualitative Comparisons↩︎

The per-benchmark galleries above use gpt-5.4-mini. To show that the improvement is not specific to one backbone, Figures 19–23 hold the benchmark and the task fixed and vary the base model: each page covers one benchmark across four models (columns) and three shared tasks (stacked blocks, the shared target shown in each strip), with the single model artifact above the PairCoder artifact for the same model and task. The chart, plot, and vector panels use tasks where both arms render, so the comparison isolates quality; the 3DCodeBench panel uses tasks where several backbones fail to render and PairCoder repairs execution (gray “no render” marks the failure).

10 P3D-Bench: Detailed Results↩︎

10.0.0.1 Full geometry breakdown.

Table 3 gives the full P3D-Bench geometry breakdown, adding the capped Chamfer score (CD) to the F-score, normal-consistency (NC), and volumetric-IoU sub-metrics already reported per model in Table ¿tbl:tab:main95grid?. All are the worst-filled mean over the 400-case text-to-3D split (invalid programs score \(0\), so the metric jointly rewards executability and fidelity). PairCoder improves every sub-metric on every model. The gains are largest exactly where the single-model baseline leaves headroom: on gpt-5.4 (\(71.5\%\) valid) F-score@0.05 rises \(0.443\rightarrow0.619\), NC \(0.435\rightarrow0.605\), and IoU \(0.226\rightarrow0.334\), while on the weak doubao-1.5-lite (\(14.0\%\) valid) every sub-metric roughly triples. Two facts isolate where the gain comes from. First, on the cases the baseline already compiles, the mean raw Chamfer distance (unnormalized, lower better) is essentially unchanged (e.g.\(0.0042\rightarrow0.0042\) on doubao-seed-2.0-mini, \(0.0029\rightarrow0.0029\) on gpt-5.4-mini): PairCoder does not perturb solids that already build. Second, the worst-filled sub-metrics nonetheless rise sharply, because PairCoder converts a large fraction of non-executable programs into valid solids that recover the target’s structure: the coverage gain, not a re-optimization of already-valid geometry, drives the numbers, consistent with the conservative-review design.

3.5pt

10.0.0.2 Topology breakdown.

Table 4 expands the P3D-Bench topology column of Table ¿tbl:tab:main95grid? into its three official sub-metrics, again as worst-filled means: the fraction of solids with no open edge (NoOE), and one minus the inverted-normal (InvN) and non-manifold-edge (NM) ratios; all lie in \([0,1]\) with higher better. The pattern mirrors geometry: PairCoder improves every topology sub-metric on every model, most strongly where the baseline leaves headroom (gpt-5.4 NoOE \(0.695\rightarrow0.970\), doubao-1.5-lite NoOE \(0.130\rightarrow0.400\)). Because a program must build a valid solid before it can be watertight or manifold, these gains are again driven by converting non-executable programs into well-formed solids rather than by repairing already-valid meshes.

16pt

10.0.0.3 Qualitative examples: geometry refinement.

On P3D-Bench text-to-3D the compile-grounded PairCoder of Table ¿tbl:tab:main95grid? gains by coverage: it repairs non-executable programs into valid solids and leaves already-compiling programs unchanged, so on the common set the two solids are identical. We additionally evaluate a vision-grounded geometry-refinement variant: when the single-model program compiles to a valid but geometrically poor solid, the Navigator renders the solid and reviews it against the specification, and the Driver refines the parametric program (kept only if it still compiles). Across a sample of \(40\) mid-quality valid solids (\(0.2<\text{F@}0.05<0.8\)) on gpt-5.4, this refinement raises F-score@0.05 on \(29\) and regresses on \(3\) (the kept-if-compiles safeguard bounds regressions), for a mean gain of \(+0.26\); it is thus a genuine effect rather than a cherry-pick. Figure 24 shows cases where both the single model and this variant build valid solids, but the refined one is much closer to the target: each triplet renders the ground-truth solid (green), the single-model solid (pink; valid but wrong), and the PairCoder-refined solid (blue), with the per-case F-score@0.05 improvement annotated.

10.0.0.4 A second Text-to-3D format: OpenSCAD.

To test whether the P3D-Bench gain is specific to the minimal-JSON format, we repeat the 400-case Text-to-3D experiment with OpenSCAD, a constructive-solid scripting language, on the three models for which both formats were run. Table 5 reports validity, the fraction of programs that compile to a valid solid, for both formats side by side. Two observations stand out. First, PairCoder drives OpenSCAD validity to a perfect \(1.000\) on all three models, repairing every non-compiling program. Second, the single-model baseline is markedly more fluent in OpenSCAD than in the minimal-JSON format (for example gpt-5.4 baseline validity \(0.715\rightarrow0.932\)), so the coverage headroom PairCoder recovers is correspondingly smaller: its OpenSCAD validity gain is \(+6.8\) points on gpt-5.4 versus \(+27.5\) points in minimal-JSON. This is the same headroom-tracking pattern seen across the rest of the grid. The gain concentrates where the baseline leaves room; OpenSCAD’s higher baseline fluency compresses it, while PairCoder still closes the remaining gap to perfect executability in both formats.

18pt

10.0.0.5 Image-to-3D and assembly-3D demonstration.

The other two P3D-Bench tracks, image-to-3D and assembly-3D, condition on a rendered image of the target (assembly-3D also on part annotations) and require a vision-capable model; their full evaluation sets build on the licensed Fusion 360 Gallery. We run the three in-repo demo cases per track with gpt-5.4 across the CadQuery and Three.js program formats (OpenSCAD is already evaluated at scale on text-to-3D, Table 5), scoring the same non-VLM buckets as text-to-3D. Table 6 reports the result. In CadQuery, PairCoder repairs the one non-compiling image-to-3D case (\(0.667\rightarrow1.000\) valid) and recovers a valid assembly-3D solid from a baseline that builds none (\(0\rightarrow0.333\)), via a sibling candidate. In Three.js, gpt-5.4 already compiles all three image-to-3D cases, so PairCoder ties, consistent with the headroom pattern seen across the grid. The three-case sample is illustrative rather than conclusive, but it shows the compiler-grounded mechanism reaches the image-conditioned tracks and a web-rendering representation, spanning all four P3D-Bench program formats together with text-to-3D.

10pt

11 Reproducibility Notes↩︎

For every benchmark we use the official task set and scorer where released (ChartMimic, Plot2Code, PandasPlotBench, StarVector, DaTikZ, 3DCodeBench, GenCAD-Code, VerilogEval, RTLLM, WebApp1K, LiveCodeBench, BigCodeBench, DS-1000, HumanEval-X), identical prompts and code extraction in both arms, and provider default sampling with thinking disabled. Visual metrics are computed with SSIM at 256\(\times\)​256 and the public CLIP, DINOv2, and SigLIP-2 checkpoints; Chamfer distance is computed on normalized surface samples, and aggregate variants assign non executing generations the worst observed score so that executability cannot be traded for similarity. Token counts are taken from provider usage fields and logged separately for the baseline arm and the PairCoder arm.

References↩︎