1 Introduction↩︎

Modern LLM systems increasingly act not by producing text alone but by producing code that is run on the user’s behalf. A growing class of systems hands the model a code interpreter: the model writes a self-contained program, an interpreter executes it, and the result is returned to the user or fed back to the model for the next step. This pattern underlies commercial tools such as Anthropic’s Claude code execution tool [1] and OpenAI’s Code Interpreter [2], as well as research systems such as CodeAct [3] and Open Interpreter [4], where executable Python program is the unit of action. In these settings the program is usually a short, single-file script that runs a piece of data analysis or produces a plot, and the user waits for the whole writing-and-running process to finish before seeing a result.

This workflow is serial, where the model first generates the complete program, and only after the final token is emitted does execution begin. While the model is generating, the interpreter sits idle. While the interpreter runs, the model has nothing left to do. The user-perceived latency is therefore the sum of two phases that run back to back, even though running an early Python statement never requires the later code to be generated first.

Our starting observation is a property of how these models write code. A human developer revises constantly, jumping back to edit earlier lines, so a program is only safe to run once the developer decides it is done. A token-based language model does not work this way. Under standard autoregressive decoding, it emits tokens from left to right and never rewrites a token once it has been produced, so every prefix of the output is already final. As soon as the model has emitted a complete statement, that statement is exactly what will appear in the finished program, and there is no reason to wait for the rest of the code before running it. We call this idea parallel execution: each statement is dispatched to the interpreter the moment it is complete, so that execution of earlier statements overlaps with generation of later ones. Figure 1 illustrates the effect. Serial execution costs roughly \(T_{gen} + T_{exec}\), whereas parallel execution costs roughly \(T_{gen} + T_{tail}\), where \(T_{\text{gen}}\), \(T_{\text{exec}}\), and \(T_{\text{tail}}\) denote the generation time, the total execution time, and the execution time of only the final chunk, respectively.

To the best of our knowledge, the parallel paradigm between the execution and generation of LLM-produced code has not yet been explored in the research community. Prior work has proposed incremental execution strategies [4], [5], where a model generates a few lines, executes them, and conditions subsequent generation on the observed output. For example, Open Interpreter [4] executes code in Jupyter-style cells and feeds the output back to the model, and EG-CFG [6] integrates real-time execution signals into the code generation process. These approaches use execution feedback to improve code quality by steering subsequent generation. However, they are still fundamentally serial with respect to latency: the model must pause generation while awaiting each execution result, and the total wall-clock time includes both generation and execution in full. In contrast, the parallel execution paradigm we propose targets a complementary goal: reducing user-perceived latency by overlapping generation and execution, without altering the generated code. As a result, the following question remains open: Is parallel execution of LLM-generated code practically viable, and what are its benefits and costs?

To realize this paradigm, we present Eager (Executing As you GEneRate), a code execution framework to support script-style code generation. We demonstrate this paradigm for Python, the dominant language in LLM code generation, though the principle applies to other interpreted languages. Its core design follows a producer–consumer pipeline, where the LLM acts as the producer and the executor as the consumer. As the language model generates tokens, an AST-based chunker incrementally identifies complete Python statements from the token stream and enqueues each chunk into a buffer. Concurrently, an executor dequeues and runs available chunks within a persistent interpreter session, preserving variable bindings across successive batches. The executor employs a dynamic batching strategy: when multiple chunks have accumulated in the queue, it merges them into a single batch, thereby amortizing the per-invocation setup overhead. Additionally, Eager features an optional early error interruption mechanism: since chunks are executed as they are generated, a runtime error in any chunk is detected immediately, at which point Eager terminates the LLM generation and returns the error along with the partially generated code. This avoids wasting generation time on code that depends on already-failed state.

We evaluate Eager across four Python benchmarks, seven LLMs, and three execution environments. The overlap mechanism hides most execution behind generation. It drives the non-overlapped portion of execution time to near zero in most settings and lowers end-to-end latency by up to 37.3% on error-free runs, with the largest gains where generation is fast relative to execution, so that execution makes up a larger share of total latency. The optional early interruption avoids wasting generation on already-failed code, and by handing the model earlier and more focused error feedback it raises subsequent repair success by up to 44.3 percentage points on data-centric benchmarks. This prevents the model from being anchored by the incorrect code generated after the error, giving it more freedom to produce a corrected solution.

We open-source our artifacts at https://doi.org/10.6084/m9.figshare.31869469. In summary, this paper makes the following contributions:

• We formalize parallel execution for LLM code interpreters and derive closed-form latency bounds that identify its speedup potential and the regimes where it pays off, including the condition under which it never regresses below serial execution, namely when the cumulative per-chunk overhead stays within the one-time serial setup cost.

• We present Eager, a Python implementation featuring AST-based chunking, gated dynamic batching, and early error interruption.

• We evaluate Eager on four benchmarks, seven LLMs, and three environments, reporting the latency benefit of overlap and the repair benefit of early interruption separately, along with the conditions under which each applies.

2 Parallel Execution↩︎

In this section, we introduce the general workflow of parallel execution and theoretically analyze its latency.

2.1 Workflow↩︎

Current LLMs follow a strictly sequential paradigm for code execution: the model first generates the entire program and only then invokes the execution environment. This results in a clear temporal separation between generation and execution, leading to significant idle time on both sides.

We instead formulate the process as a streaming pipeline. The LLM acts as a producer that autoregressively emits tokens. A detection module continuously processes the token stream to identify executable chunks, defined as minimal syntactically complete and semantically executable units. Once a chunk is detected, it is immediately dispatched to an execution engine, which maintains a persistent session to preserve program state. This design enables temporal overlap across all stages: while the LLM generates tokens for later chunks, earlier chunks can already be detected and executed.

2.2 Theoretical Modeling↩︎

We model the system as a three-stage pipeline consisting of generation, detection, and execution. The total latency is determined by the critical path through these stages.

Notation. We define:

• \(L\): total number of tokens,

• \(v_{gen}\): generation speed (tokens per second),

• \(T_{FT}\): time-to-first-token,

• \(N\): number of executable chunks,

• \(l_i\): length of chunk \(i\), with \(\sum_{i=1}^N l_i = L\),

• \(\delta_i\): residual detection delay for chunk \(i\) (the portion of detection cost not hidden behind generation),

• \(T_{setup}\): per-chunk execution overhead,

• \(T_{exe,i}\): execution time of chunk \(i\).

2.2.1 Serial Execution↩︎

In the serial paradigm, the model first generates the complete program, after which the interpreter executes it as a monolithic block. The total latency is therefore: \[T_{serial} = T_{FT} + \frac{L}{v_{gen}} + T_{setup}^{(\mathrm{full})} + T_{exe}^{(\mathrm{full})},\] where \(T_{setup}^{(\mathrm{full})}\) denotes the one-time execution setup cost for the complete program, and \(T_{exe}^{(\mathrm{full})}\) denotes the execution time of the full program.

For consistency with the chunked formulation, one may approximate: \[T_{exe}^{(\mathrm{full})} \approx \sum_{i=1}^{N} T_{exe,i},\] but importantly, the serial baseline does not incur repeated per-chunk setup overhead or streaming detection overhead.

2.2.2 Parallel Execution↩︎

In the parallel paradigm, generation, detection, and execution are overlapped. Let \(t_{g,i}\) denote the time when chunk \(i\) has been fully generated, and let \(t_{e,i}\) denote the time when execution of chunk \(i\) completes.

Generation. The time at which chunk \(i\) has been fully generated depends on the cumulative token length of all preceding chunks: \[t_{g,i} = T_{FT} + \frac{\sum_{j=1}^{i} l_j}{v_{gen}}\]

Detection. The detector processes the token stream online. We write the chunk-ready time as: \[t_{d,i} = t_{g,i} + \delta_i\] where \(\delta_i\) is the residual detection delay, i.e., the portion of detection cost that is not hidden behind generation.

Execution. Chunk \(i\) can only start executing once it has been detected from the generated code and the previous chunk has finished executing: \[t_{e,i} = \max(t_{d,i},\, t_{e,i-1}) + T_{setup} + T_{exe,i}\]

The overall parallel latency is the completion time of the final chunk: \[T_{parallel} = t_{e,N}\]

Closed-form characterization. Unrolling the recurrence gives \[T_{parallel} = \max_{1 \le i \le N} \left[ t_{g,i} + \delta_i + \sum_{j=i}^{N} (T_{setup} + T_{exe,j}) \right]\]

2.2.3 Latency Bounds↩︎

The closed-form expression of the parallel execution allows us to derive upper and lower bounds on the latency.

Upper bound. For any \(i\in \{1, \ldots, N\}\), the generation prefix satisfies \(\sum_{j=1}^{i} l_j \le L\), the detection residual satisfies \(\delta_i \le \bar{\delta} \triangleq \max_{1 \le k \le N} \delta_k\), and the execution tail satisfies \(\sum_{j=i}^{N}(T_{setup}+T_{exe,j}) \le \sum_{j=1}^{N}(T_{setup}+T_{exe,j})\). Applying these to every term inside the outer maximum yields

\[\label{eq:upper} T_{parallel} \;\le\; T_{FT} + \frac{L}{v_{gen}} + \bar{\delta} + N\,T_{setup} + \sum_{j=1}^{N} T_{exe,j}\tag{1}\]

This upper bound corresponds to a zero-overlap execution in which every stage waits for its predecessor to complete entirely. Compared with the serial baseline, the additional cost is at most \[\label{eq:overhead} T_{parallel} - T_{serial} \;\le\; \bar{\delta} + N\,T_{setup} - T_{setup}^{(\mathrm{full})}\tag{2}\] , which captures the overhead from (i) streaming detection and (ii) repeated per-chunk setup. In practice \(\bar{\delta}\) is on the order of milliseconds because the detector operates on a lightweight grammar, so the overhead is dominated by the cumulative setup cost \(N\,T_{setup} - T_{setup}^{(\mathrm{full})}\). When both \(\bar{\delta}\) and \(N\,T_{setup} - T_{setup}^{(\mathrm{full})}\) are negligible, the upper bound reduces to \(T_{serial}\), showing that the parallel scheme introduces no regression under these conditions.

Lower bound. Two structural constraints yield complementary lower bounds.

1. Generation constraint (setting \(i = N\)). The system must generate all tokens before the last chunk can complete: \[\label{eq:lb-gen} T_{parallel} \;\ge\; T_{FT} + \frac{L}{v_{gen}} + \delta_N + T_{setup} + T_{exe,N}\tag{3}\]

2. Execution constraint (setting \(i = 1\)). The system must execute all chunks in order after the first chunk becomes available: \[\label{eq:lb-exec} T_{parallel} \;\ge\; T_{FT} + \frac{l_1}{v_{gen}} + \delta_1 + N\,T_{setup} + \sum_{j=1}^{N} T_{exe,j}\tag{4}\]

Combining both gives the composite lower bound: \[\label{eq:lb} T_{parallel} \;\ge\; \max\!\left\{ \begin{align} & T_{FT} + \frac{L}{v_{gen}} + \delta_N + T_{setup} + T_{exe,N},\\ & T_{FT} + \frac{l_1}{v_{gen}} + \delta_1 + N\,T_{setup} + \textstyle\sum_{j=1}^{N} T_{exe,j} \end{align} \right\}\tag{5}\] The first term dominates when generation is the bottleneck (generation-dominated regime); the second dominates when execution is the bottleneck (execution-dominated regime).

2.2.4 Speedup Bounds↩︎

Define the speedup \(S = T_{serial} / T_{parallel}\). From the lower bound 3 on \(T_{parallel}\), the speedup is at most: \[\label{eq:speedup-ub} S \;\le\; \frac{T_{FT} + \frac{L}{v_{gen}} + T_{setup}^{(\mathrm{full})} + T_{exe}^{(\mathrm{full})}}{T_{FT} + \frac{L}{v_{gen}} + \delta_N + T_{setup} + T_{exe,N}}\tag{6}\] when \(\delta_N\) and \(T_{exe,N}\) are small relative to \(L/v_{gen}\), which simplifies to \(S \lesssim 1 + (T_{setup}^{(\mathrm{full})} + T_{exe}^{(\mathrm{full})}) / (T_{FT} + L/v_{gen})\). This is achieved when all execution is hidden behind generation. From the upper bound 1 , \(S \ge 1\) whenever \(T_{setup}^{(\mathrm{full})} \ge \bar{\delta} + N\,T_{setup}\), i.e., the one-time serial setup cost exceeds the cumulative chunk overhead. Whether this condition holds depends on the execution environment. In our experimental setup, where chunks are dispatched to a persistent REPL session with per-call overhead on the order of 1 ms, \(T_{setup}\) and \(\bar{\delta}\) are both small enough for the condition to be satisfied comfortably. However, in environments with heavier per-chunk orchestration costs (e.g., container cold starts or cross-process IPC), the cumulative term \(N\,T_{setup}\) may become non-negligible, and the condition should be verified empirically.

2.2.5 Regime Analysis under Uniform Chunks↩︎

To obtain sharper insight, we analyze the special case of uniform chunks: \(l_i = L/N\), \(T_{exe,i} = \tau_e\), and \(\delta_i = \delta\). Define the per-chunk generation time \(\alpha \triangleq L/(N\,v_{gen})\) and per-chunk execution time \(\beta \triangleq T_{setup} + \tau_e\). The inner term of the closed-form expression becomes affine in \(i\): \[f(i) = \underbrace{T_{FT} + \delta + (N+1)\beta}_{\text{constant}} + i\,(\alpha - \beta)\] yielding the following three regimes:

R1: Generation-dominated (\(\alpha > \beta\)). The maximum is at \(i = N\): \[\label{eq:gen-dom} T_{parallel} = T_{FT} + \frac{L}{v_{gen}} + \delta + \beta\tag{7}\] Execution of every chunk except the last is entirely hidden behind generation.

R2: Execution-dominated (\(\alpha < \beta\)). The maximum is at \(i = 1\): \[\label{eq:exec-dom} T_{parallel} = T_{FT} + \alpha + \delta + N\beta\tag{8}\] Generation of every chunk except the first is hidden behind execution.

R3: Balanced (\(\alpha = \beta\)). The pipeline is perfectly paced. Setting \(\alpha = \beta\), i.e., \(L/(N\,v_{gen}) = T_{setup} + T_{exe}^{(\mathrm{full})}/N\), and solving for \(N\) gives the critical chunk count: \[\label{eq:nstar} N^* = \frac{L/v_{gen} - T_{exe}^{(\mathrm{full})}}{T_{setup}}\tag{9}\] which is positive whenever total generation time exceeds total execution time, the common case for LLM code generation. For \(N \le N^*\) additional chunks improve overlap; beyond \(N^*\) the cumulative setup overhead \(N\,T_{setup}\) dominates and latency degrades.

3 Implementation↩︎

Building on the theoretical framework in Section 2.2, we present Eager, a concrete implementation of parallel execution for LLM code generation. Eager instantiates the pipeline with design choices aimed at minimizing the detection and per-chunk setup overhead identified in Section 2.2, without affecting execution outcomes.

As illustrated in Figure 2, Eager consists of a chunker and an executor. Chunker accumulates the streaming tokens generated by LLMs in a buffer and identifies complete Python statements via AST parsing. Detected chunks are dispatched to a pending queue, from which an executor dequeues, applies gating and batching optimizations, and delegates execution to a persistent session. If a runtime error is encountered, the executor sends an interrupt signal back to terminate the LLM generation immediately (the dashed path in Figure 2). We describe each component in detail below.

3.1 Producer: AST-Based Chunker↩︎

The chunker operates on the streaming token output from an LLM and identifies executable chunks incrementally. We implement the chunker based on AST statement boundaries. As each token arrives, it is appended to a code buffer, and the chunker attempts to parse the buffer into a Python AST. A chunk boundary is recognized when the buffer forms a complete top-level statement (e.g., assignments, loops, or function calls). A statement is considered complete when it can be unambiguously determined that no further tokens will be generated as part of it, i.e., the statement has no remaining portions yet to be produced by the LLM. Once a chunk is confirmed as complete, the corresponding statements are removed from the buffer and dispatched to the executor, while any remaining tokens stay in the buffer for subsequent detection.

In many cases, syntactic completeness directly implies completeness. For example, upon receiving print("hi") followed by a newline, the chunker can immediately confirm this as a standalone statement and dispatch it. However, in other cases, a syntactically valid statement may still have remaining portions to be generated. Consider generating a function where tokens arrive incrementally. After receiving the first line of the function body, the parser can already yield syntactically valid results. Yet the next line could continue the function body with additional statements, in which case the definition is not yet complete. To handle such ambiguities, Eager employs a lookahead strategy: when a statement parses successfully but its completeness cannot be definitively confirmed from syntax alone, the chunker waits for one additional token before committing. If the next token rules out the possibility of the current statement continuing (e.g., by beginning a new top-level statement or indicating a dedent), the chunk is finalized and dispatched; otherwise, the buffer continues to accumulate. This mechanism ensures that dispatched chunks are both syntactically complete and semantically independent.

3.2 Consumer: Gated Executor with Dynamic Batching↩︎

The executor acts as the consumer of the pipeline, receiving chunks dispatched by the chunker. It interacts with a persistent execution session, whether a local Python subprocess, a sandboxed environment, or a Docker container, that preserves all imports, variable bindings, and function definitions across successive chunk executions. This execution session is therefore responsible for the execution of the chunks.

Confirmed chunks are placed into a pending queue. When the executor becomes available, it merges all currently pending chunks into a single big chunk, rather than executing them one by one. This dynamic batching naturally adapts to the pace difference between the chunker and the executor: when execution is slower than detection, more chunks accumulate and are batched together, reducing the effective number of executor invocations and thus the cumulative setup cost \(N\,T_{setup}\) in Equation 1 . This also makes the speedup condition \(S \ge 1\) easier to satisfy, since the overhead term \(N\,T_{setup} - T_{setup}^{(\mathrm{full})}\) in Equation 2 shrinks accordingly.

On top of batching, the executor applies a gating policy that classifies each chunk before dispatch. The policy distinguishes four categories. The first identifies chunks that are unprofitable to execute eagerly and defers them within the pipeline. The remaining three identify chunks whose Eager execution would be unsafe or would alter program semantics, and fall back to serial execution for them.

• Low-yield chunks. Chunks whose execution time is negligible compared to the per-invocation setup overhead yield no useful overlap. Typical examples are function or class declarations, which produce no observable computation when executed in isolation. For such chunks, the setup cost \(T_{setup}\) dominates the actual execution time \(T_{exe,i} \approx 0\), so dispatching them individually only adds overhead. The executor therefore defers these chunks: they remain in the pending queue and are merged with the next non-deferred chunk, at which point the declarations become available to support subsequent code that depends on them.

• External-side-effect chunks. Eager execution changes when a statement runs relative to generation, not whether it runs. For most statements this reordering is harmless, but operations with irreversible, externally observable effects must not be committed before the surrounding code has been generated. For chunks invoking process creation, filesystem mutation, network communication, or inter-process operations (e.g., os, subprocess, shutil, socket, multiprocessing), Eager disables overlap and executes the program serially, so that their side-effect timing matches the serial baseline.

• Timing-sensitive chunks. Some statements have semantics that depend on wall-clock timing or on the surrounding execution schedule, such as sleeping, timers, clock reads, signal delivery, and thread creation (e.g., time, signal, threading). Executing such chunks ahead of generation would perturb a program’s own timing or concurrency assumptions, altering its observable behavior. Eager therefore falls back to serial execution for these programs, keeping such statements in program order and preserving correctness for code that observes its own timing.

• Dynamic-execution chunks. Dynamic execution entry points (eval, exec, compile, __import__) make a chunk’s behavior undeterminable from its surface syntax, so the gating policy cannot reason about which of the categories above it belongs to. Rather than dispatch code whose effects cannot be statically inspected, Eager conservatively falls back to serial execution.

The gate is applied as a static, name-based check over each chunk’s parsed AST. Its default instance is a conservative denylist of standard-library modules that commonly carry external effects, and a deployer can extend this set for a given workload.

3.3 Error Handling in the Pipeline↩︎

The preceding subsections describe the normal flow of the pipeline: the chunker detects and dispatches chunks, and the executor batches and executes them. We now describe how the pipeline handles runtime errors.

In serial execution, runtime errors are only observed after the entire program has been generated and executed as a whole. In Eager, since chunks are executed incrementally alongside generation, a runtime error is caught as soon as the failing chunk finishes execution. At that point, Eager immediately terminates the LLM generation process. The error message, together with the code generated up to the point of failure, is returned to the caller. No further tokens are generated or executed.

This early interruption provides two benefits. First, it avoids spending generation time on code that depends on already-failed state, effectively reducing both the token count \(L\) and chunk count \(N\) on failure paths and thus lowering end-to-end latency. Second, it delivers error feedback at the earliest possible point, enabling a tighter repair loop—as we empirically validate in RQ3 (Section 5.4), this earlier feedback leads to higher repair success rates in most scenarios.

Notably, this early interruption is an optional feature. Eager can also be configured to continue generation after an error is detected and defer the error report until the full program has been produced. Under this configuration, Eager still provides latency savings through overlapped execution of the error-free prefix, while preserving the same complete-code error semantics as serial execution.

4 Experiment Setup↩︎

Using Eager as a demonstration, we experimentally assess the performance of parallel execution. In this section, we will introduce the settings of the experiments, including the benchmarks, LLMs, execution environments, and implementation details. The rationale behind our setup is driven by answering the following three research questions:

• RQ1: How much latency can Eager save compared to serial execution across different token generation speeds and execution environments?

• RQ2: Do the latency savings generalize from simulated to real LLM code generation?

• RQ3: Does the earlier error feedback provided by Eager help or hinder LLMs’ subsequent code repair?

4.1 Benchmarks↩︎

We use four benchmarks where the LLMs are required to produce executable Python scripts.

• DSBench [7] and DABench [8] are both benchmarks that evaluate LLMs on data analysis tasks, where the model is given tabular data files along with natural language questions and must generate executable Python scripts to derive the answers. DSBench is sourced from Modeloff financial analysis competitions and Kaggle challenges, featuring 442 data analysis tasks with realistic settings such as long task descriptions, multimodal backgrounds, and multi-table structures. DABench provides 257 data analysis questions derived from real-world CSV files crawled from GitHub.

• PandasPlotBench [9] (PdPlotBench) is a human-curated benchmark for evaluating LLMs’ ability to generate data visualization code. It contains 175 tasks, each requiring the model to produce plotting code given a natural language description and a Pandas DataFrame specification.

• GitChameleon [10] is a manually curated benchmark that evaluates LLMs’ ability to generate version-specific Python code. It contains 116 code completion problems, each conditioned on a particular library version and accompanied by executable unit tests. Unlike conventional code generation benchmarks, GitChameleon specifically tests whether models can produce code compatible with a specified version of a library, reflecting real-world scenarios where developers are constrained to specific dependency versions.

4.2 Large Language Models↩︎

We evaluate Eager across seven LLMs spanning both open-weight and proprietary models, covering a range of model scales, speeds, and architectures.

For open-weight models, we use DeepSeek-V3.2 [11], a 685B-parameter MoE model with sparse attention; MiMo-V2-Flash [12], Xiaomi’s 309B MoE model (15B active) optimized for fast inference via hybrid sliding-window attention and multi-token prediction; Qwen3-Coder [13], Alibaba’s 480B MoE coding model (35B active) trained with long-horizon reinforcement learning for agentic coding; and DeepSeek-Reasoner [14], a reasoning-focused model that employs extended chain-of-thought during inference.

For proprietary models, we use GPT-4o-mini [15], OpenAI’s lightweight multimodal model; GPT-5.1-Codex-Mini [16], a variant of GPT-5.1-Codex further optimized for agentic coding tasks; and Gemini-3.1-Flash-Lite [17], Google’s model in the Gemini 3 series designed for high-volume, low-latency workloads.

4.3 Execution Environments↩︎

As the LLM-generated code could be executed in various environments in practice, we experiment with three mainstream options:

• Local execution runs the generated Python scripts directly on the host machine, offering the lowest overhead and fastest startup time. This represents the simplest setup commonly used in lightweight scripting and prototyping workflows.

• Docker-based execution runs the scripts inside an isolated Docker container [18]. This provides a reproducible and sandboxed environment that prevents unintended side effects on the host system, and is widely adopted in production-grade agent frameworks for dependency management and security.

• Open Interpreter sandbox executes code through the Open Interpreter [19] framework, which provides a managed sandboxed environment with built-in support for LLM-driven code execution. It represents a higher-level abstraction where the execution runtime is integrated into an end-to-end agent pipeline.

These three environments allow us to evaluate whether the latency savings of Eager generalize across practical deployment settings. All environments are configured with 2 CPU cores to reflect the resource-constrained settings typical of cloud-hosted code execution sandboxes.

4.4 Evaluation Metrics↩︎

We use two metrics to quantify the latency impact:

• Non-overlapped Execution Latency (NEL) measures the portion of code execution time that falls outside the LLM generation phase. In serial execution, the entire execution time is non-overlapped, as execution begins only after generation completes. In parallel execution with Eager, part of the execution overlaps with the ongoing token generation, and this metric captures only the remaining portion that still contributes to user-perceived delay. A lower non-overlapped execution time indicates that more execution has been effectively hidden behind generation.

• End-to-End Latency (E2EL) measures the wall-clock time from the start of the LLM call to the completion of code execution, i.e., the total time the user must wait. It equals the sum of the LLM generation time and the non-overlapped execution time. This metric directly reflects the delay perceived by the user.

4.5 Implementation Details↩︎

All experiments are conducted on a server running Ubuntu 22.04 with an Intel Xeon Platinum 8352V processor. To ensure reliable and reproducible timing measurements, we apply single-CPU isolation via taskset for local and Open Interpreter runs, and CPU pinning (--cpuset-cpus) for Docker-based runs. For code execution, the local and Docker environments use a persistent Python REPL subprocess that preserves imports, variables, and definitions across chunks, with per-call overhead of approximately 1 ms. The Open Interpreter environment uses its native Jupyter-kernel backend as the executor. All experiments use isolated, task-local executors to prevent cross-task interference. The LLMs are accessed through OpenRouter APIs [20] via streaming mode. All model names used throughout the paper correspond to their official API identifiers as of March 2026.

5 Results↩︎

5.1 Preliminary: Chunk Reconstruction Fidelity↩︎

Before measuring the latency benefits of Eager, we first validate that its chunking mechanism preserves program integrity. We collect the code generated by all seven LLMs across the four benchmarks (DABench, DSBench, PandasPlotBench, and GitChameleon) and run each program through Eager’s chunking pipeline under the Docker environment. For every program, we concatenate the emitted chunks and compare the result against the original generated code. Across all programs and all seven models, the reassembled code is character-level identical to the original in every case. This confirms that the AST-based chunker correctly identifies statement boundaries without dropping, duplicating, or reordering any tokens, regardless of the generating model.

Since Eager executes chunks in sequence within a single persistent session, carrying forward all imports, variable bindings, and function definitions across chunk boundaries without any concurrency or re-initialization, lossless reconstruction directly implies execution equivalence for deterministic programs, which constitute all the LLM-generated code on our benchmarks.

3pt

5.2 RQ1: Latency Savings Across Generation Speeds and Environments↩︎

As the generation speed of real LLMs are hard to systematically control, we choose a simulated setting. Instead of using real LLMs to produce the token stream for Eager to execute, we use the already generated code solutions from all four benchmarks and replay them as a mock token stream at a fixed Token-Per-Second (TPS) rate. Specifically, we use the gold (reference) solutions from PandasPlotBench and GitChameleon and the runnable solutions generated by DeepSeek-V3.2 for DABench and DSBench. We sweep across four representative TPS rates—20, 50, 100, and 200—covering the range from slower open-weight model deployments to fast proprietary APIs. We run all these code scripts across the three execution environments (local, Docker, and Open Interpreter) and compare Eager against serial execution in terms of non-overlapped execution time and end-to-end latency. As shown in Table 1, Eager achieves consistently high NEL savings across all benchmarks, environments, and TPS rates, averaging 89.8%. This indicates that the vast majority of execution time is successfully hidden behind generation regardless of the specific configuration. For example, on DABench and DSBench, the NEL savings remain above 89% even at 200 TPS across all three environments, meaning that only about 10% of the original execution time is left exposed after overlapping. GitChameleon reaches 91.1% NEL savings at 20 TPS under Docker, indicating that execution is largely hidden behind generation at lower generation speeds. The E2EL savings, while more modest, follow a clear trend: they increase as generation speed decreases. At 20 TPS, end-to-end savings are relatively small (1–6% across benchmarks) because generation time dominates and the absolute execution time being hidden is small relative to the total. At 200 TPS, the savings grow substantially (up to 34.9% on PandasPlotBench under Docker), as faster generation compresses the generation window and makes execution a larger fraction of the total latency. This is consistent with the theoretical prediction that the E2EL benefit of parallel execution scales with the ratio of execution time to generation time. Across the three environments, the savings are broadly consistent, confirming that Eager generalizes across deployment settings. Docker and Local show similar results, with Docker having a slight edge due to more stable timing from CPU pinning. Open Interpreter exhibits marginally lower savings, particularly on PandasPlotBench and GitChameleon at higher TPS rates (e.g., 73.3% NEL on PandasPlotBench at 200 TPS, but% under Docker), which we attribute to the additional overhead of the Jupyter-kernel execution backend.

5.3 RQ2: Generalization to Real LLM Code Generation↩︎

To validate whether the savings observed in RQ1 generalize to real-world settings, we use each of the seven LLMs to generate code for the four benchmark tasks via streaming. The generated code is executed with Eager in the Docker environment, and the resulting latency is compared against the serial execution baseline. Notably, when a runtime error occurs during execution, Eager interrupts the code generation process immediately. We therefore report results separately for error-free and error-encountered executions. As shown in Table 2, Eager consistently reduces latency across nearly all model–benchmark combinations under both conditions. For error-free executions, the non-overlapped execution latency (NEL) drops to near zero in many cases (e.g., 2 ms for DeepSeek-V3.2 on DABench, 1 ms for GPT-4o-mini on DABench), indicating that execution is almost entirely hidden behind generation for these settings. The end-to-end latency (E2EL) improves correspondingly, with reductions ranging from modest savings on benchmarks where execution is already fast (e.g., DSBench with GPT-5.1-Codex, where the baseline NEL is only 62 ms) to substantial reductions where execution constitutes a larger fraction of total latency (e.g., Gemini-3.1 on PandasPlotBench, from 1440 ms to 903 ms, a 37.3% reduction). The NEL savings are largest for slower models, whose longer generation window leaves more room to hide execution. The E2EL savings move the other way: they are largest for faster models such as Gemini-3.1, where the shorter generation window makes execution a bigger share of total latency. This matches the speedup bound in Section 2.2, where the end-to-end benefit grows as generation time falls relative to execution. For error-encountered executions, Eager provides even larger latency reductions, as the early interruption mechanism eliminates both the remaining generation time and the execution of subsequent code that would inevitably depend on already-failed state. This is most visible in the E2EL: Qwen3-Coder on DSBench drops from 13163 ms to 3289 ms, and DeepSeek-R on PandasPlotBench achieves a 55.3% reduction. The NEL for error cases is often 0 ms on benchmarks like DABench and PandasPlotBench, meaning the error was caught entirely within the generation window. For GitChameleon error cases, the NEL under Eager is undefined because generation is terminated at the point of error, so only E2EL is reported. The E2EL nonetheless shows clear improvements across all models (e.g., DeepSeek-R from 9994 ms to 5247 ms), confirming that early interruption saves substantial waiting time even when the post-generation tail cannot be measured.

5.4 RQ3: Effect of Earlier Error Feedback on Code Repair↩︎

As observed in RQ2, Eager catches errors during generation rather than after it completes, providing earlier feedback to the LLM. This early interruption also terminates the code generation process, leaving the remaining code ungenerated. In this RQ, we investigate whether the partial code resulting from early error interruption affects the model’s ability to resolve the detected error in subsequent repair attempts. Specifically, for each error-encountered sample in RQ2, we append the error message to the already-generated code and let the same model continue generating a fix. We compare two conditions (Full and Partial): under serial execution, the model continues with the complete code followed by the error message; under Eager, the model continues with the partially generated code followed by the error message. We then compare the error resolution rates between the two conditions—that is, whether the repaired code executes without reproducing the original runtime error. As reported in Table 3, across the three data-centric benchmarks (DABench, DSBench, and PandasPlotBench), repairing from partial code consistently achieves higher error resolution rates than repairing from full code, with improvements ranging from +2.1 to +44.3 percentage points. The gains are particularly pronounced for models such as Qwen3-Coder and Gemini-3.1, which achieve over 30 percentage points of improvement on DABench and GPT-4o-mini, which gains +44.3 percentage points on DSBench. This result may seem counterintuitive at first, as one might expect that having the complete code would provide the model with more context for repair. However, we hypothesize that the full program, which has already executed to completion and failed, may anchor the model toward preserving its original (flawed) logic. In contrast, the partial code from Eager leaves the remainder ungenerated, giving the model more freedom to regenerate a corrected solution from the point of failure. Moreover, we observe that on the three data-centric benchmarks, errors tend to occur early in the generated code, at a median position of around 35% of the total code length. This means that under serial execution, on average 65% of the code is generated after the error has already occurred, without any awareness of the runtime failure. This post-error suffix often depends on state that no longer holds (e.g., referencing a DataFrame column that caused a KeyError), providing the model with misleading context during repair. By contrast, Eager interrupts generation at the point of failure, avoiding this misleading suffix entirely. The exception is GitChameleon, where repairing from partial code slightly underperforms in most cases. We attribute this to the nature of GitChameleon tasks: they are version-specific code completion problems where the error typically stems from using an incorrect API for the target library version. In these tasks, the code after the error point often encodes the intended API usage, which is precisely the information needed to diagnose a version mismatch. To investigate further, we analyzed the 27 cases (across all models) where full-code repair resolved the error but partial-code repair did not: 63% of these cases had their code truncated by early interruption (tokens saved more than 0%), confirming that the lost suffix contained error-relevant context. The remaining 37% showed identical input code, where the difference is attributable to LLM repair stochasticity.

6 Threats to Validity↩︎

Language generalizability. Our implementation and evaluation target Python. This is a natural rather than arbitrary choice, since script-style generation in interpreted languages is the dominant setting for the code-interpreter and code-as-action systems that parallel execution targets (Section 1). The pipeline itself is language-agnostic, and only the execution backend depends on a REPL-style interpreter, so extending to other interpreted languages such as JavaScript or R is straightforward. Compiled languages would require a different backend (e.g., incremental compilation or JIT-based evaluation), which raises execution-model questions, such as whether compilation can itself be overlapped with generation, that are orthogonal to the scheduling mechanism studied here. We leave this to future work. Regardless of how that backend is realized, Equation 2 bounds the worst case: parallel execution does not regress below serial execution as long as the cumulative per-chunk overhead stays within the serial setup cost, the condition met in our REPL setup, where we observe no regression in any configuration.

Benchmark representativeness. Our method targets single-file scripts, which are the intended application domain of EAGER and the dominant workload of LLM code interpreters in both industry and research. This is a deliberate scope rather than an incidental restriction: parallel execution yields its largest benefit precisely in this generation-dominated regime (Section 2.2), where generation time dominates lightweight per-statement execution. Tasks dominated by heavy execution, such as repository-scale agent tasks that compile a project or run a full test suite, fall into the execution-dominated regime and lie outside EAGER’s intended scope. By Equation 2 , EAGER does not regress below serial execution in that regime as long as the same overhead condition holds, but its latency savings are small there, so we do not target such settings.

Docker cold-start outliers. In the Docker environment, the tasks in GitChameleon require creating isolated virtual environments for version-specific library testing. A small number of tasks (46 out of the 990 tasks across all four benchmarks) exhibited abnormally high execution times due to virtual environment cold-start overhead. We exclude these outliers from our reported results. Removing this filter changes the aggregate results by less than 1 percentage point, confirming that our conclusions are robust to this decision.

7 Related Work↩︎

Code Generation with Execution in the Loop Recent LLM work has used code execution as an important mechanism for reasoning and interaction. For example, program-aided reasoning methods such as PAL [21] and Program of Thoughts [22] usually let the model first generate a complete program and only then execute it, using code mainly as a reliable substrate for arithmetic or symbolic computation rather than as a source of fine-grained feedback during decoding . Related language-to-code approaches in the same vein also largely follow this generate-then-execute pattern [23]. Follow-up work preserved this basic structure while making the outer loop more execution-aware. LEVER uses execution outcomes to verify and rerank candidate programs [24], while Chain of Code augments executable code with selective emulation when some steps cannot be directly executed [25]. In parallel, self-correction systems such as Self-Debugging [26], CYCLE [27], and runtime-verification-based debugging methods [28] treat execution errors, failed tests, or traces as signals for iterative repair after an initial solution has already been produced. More recently, EG-CFG [6] tests the code line-by-line as it is being written, using real-time execution feedback to steer the model toward functionally correct and executable solutions. Our work is orthogonal to these efforts: rather than improving code quality, we reduce user-perceived latency by overlapping generation and execution, a technique that composes naturally with them.

Execution Environments for LLMs Recent LLM systems have also developed increasingly capable execution substrates, including interactive interpreters, notebook-style runtimes, and containerized sandboxes that make generated code executable, stateful, and reproducible across turns. OpenCodeInterpreter integrates code generation with execution and iterative refinement in a code-interpreter-style workflow [29], while InterCode exposes execution as part of the task environment through self-contained Docker sandboxes and standardized feedback channels [5]. Concurrently, infrastructure-oriented work such as MPL-Sandbox emphasizes practical multi-language isolation and unified compiler or runtime feedback for LLM-based coding systems [30], and notebook-centric agents and benchmarks further show the importance of maintaining persistent execution state in data analysis settings [31], [32]. These systems make execution available to the model, but their main goal is to provide a reliable runtime for interaction, evaluation, or iterative repair, rather than to minimize end-to-end response time. In contrast, our focus is not on building a richer sandbox or stronger debugging loop, but on a systems-level scheduling question: how to overlap generation and execution so that runtime feedback can be exploited without forcing the user to wait for a strictly sequential generate-then-run pipeline.

8 Discussion↩︎

Implications for programming language design. Existing programming languages were designed under the assumption that source code is written in its entirety before execution. However, LLM-based code generation fundamentally changes this assumption: code is produced token by token as a stream, with each prefix potentially forming a meaningful partial program. This mismatch forces systems like Eager to reconstruct executability from a stream that the language was never designed to support incrementally—relying on AST parsing heuristics, lookahead strategies, and gating policies to determine when a partial program is safe and worthy to execute. As LLM-generated code becomes an increasingly prevalent mode of program creation, we see an opportunity for future programming languages to treat streamability as a first-class design goal: for instance, through explicit statement delimiters that eliminate boundary ambiguity, or through language-level primitives that allow the runtime to consume and execute code incrementally as it is produced. Such designs would reduce the complexity of systems like Eager and more broadly benefit the emerging ecosystem of AI-assisted programming.

Preserving execution semantics under overlap Eager dispatches code before the full program is available, which raises a natural concern: running statements ahead of generation may change the program behavior that serial execution would not? Eager mitigates this issue through its gating policy in Section 3.2. It routes statements whose observable behavior may depend on when they run, such as those with sensitivity to wall-clock timing and scheduling, onto the serial path, where they are committed only after all program has been generated. Overlap is therefore confined to computation that is insensitive to its position relative to generation, where reordering cannot produce an externally observable consequence absent under serial execution. In this sense, Eager preserves the observable semantics of the serial baseline.

It is worth emphasizing that Eager is a scheduling layer deciding when a statement runs relative to generation. It is not an isolation mechanism and does not govern what generated code is permitted to do. Confining untrusted code remains the responsibility of the execution environment, whether a container, a restricted subprocess, or a managed sandbox, exactly as in the serial setting. Because effect-bearing statements retain their serial timing, an environment that behaves correctly under serial execution behaves identically under Eager: it inherits whatever isolation the deployment already provides without weakening it. The gating check itself is a static, conservative over-approximation, and any statement it cannot classify as position-insensitive runs serially. We treat the completeness of that classification as separate from the latency mechanism studied here.

9 Conclusion↩︎

In this paper, we introduced parallel execution, a paradigm that dispatches LLM-generated code statements to the interpreter as they are produced, overlapping generation and execution to reduce end-to-end latency. We formalized this paradigm theoretically and realized it in Eager, a framework featuring AST-based chunking, dynamic batching, and early error interruption. Experiments across four benchmarks, seven LLMs, and three execution environments showed that Eager consistently hides the majority of execution time behind generation, achieving end-to-end latency reductions of up to 37.3%, while the early error interruption mechanism improved error resolution rates by up to 44.3 percentage points

References↩︎

---

1. These authors contributed equally to this research.↩︎