1 Introduction↩︎

The rapid evolution of Large Language Models (LLMs) has led to their widespread adoption in automated test generation tasks [1]–[6]. Recent advances have further enabled the emergence of coding agents such as Codex CLI[7], Claude Code [8], Gemini-cli [9], and Qwen Code [10]. Coding agents are terminal-based, LLM-driven systems capable of iteratively interacting with software projects through actions such as searching, navigating, editing, building, and testing. Unlike LLM prompting approaches [1]–[6], which rely on heuristic, rule-based generation strategies, coding agents operate autonomously over multiple iterations, adapting their behaviour based on intermediate feedback from compilers, test execution outcomes, and the surrounding source code context.

Despite this autonomy, state-of-the-art coding agents suffer from critical limitations when applied to comprehensive test generation. A primary challenge is the phenomenon of lazy generation [11]. Even when explicitly instructed to maximize code coverage, agents naturally gravitate toward simple methods and straight-line execution paths, systematically avoiding complex logic that requires deep program analysis. To illustrate this, 1 demonstrates a real-world interaction with the Gemini-cli agent tasked with generating tests for Apache Commons CLI’s DefaultParser class from Defects4J [12]. Despite the agent’s confident report that tests are “generated and verified", it prematurely terminates its execution, leaving branch coverage stalled at a mere 31%.

None

To address premature termination, developers often remain in the loop to steer coding agents, deciding whether to retry, provide targeted guidance, or halt the process. These decisions rely heavily on human intuition and can either leave critical branches untested or waste computational resources on ineffective iterations. Consequently, monitoring agent progress and providing corrective feedback becomes a bottleneck that undermines the efficiency gains of automated test generation.

In this paper, we introduce Scate(Supervisor-Copiloted Agentic Test Generation Engine), an adaptive supervisor designed to replace human-in-the-loop supervision. Unlike traditional automated testing frameworks that micro-manage generation at the granular level of individual methods or execution paths [1]–[3], Scate employs a macro-level, class-centric strategy. By framing supervision as a contextual bandit problem [13], a sequential decision-making framework that learns to select actions based on the current context and observed rewards, Scate dynamically combines runtime coverage feedback with structural metrics to provide continuous, data-driven guidance that directs coding agents toward comprehensive unit test generation.

In this paper, we make the following contributions:

• Scate, to the best of our knowledge, the first framework to provide automated, adaptive supervision for coding agents in test generation. Scate is agent-agnostic, readily integrates with diverse coding agents.

• A novel formulation of agent supervision as a contextual bandit problem. By learning from static class testability and real-time code coverage metrics, the supervisor dynamically adapts its policy to maximize code coverage while minimizing computational costs.

• An empirical evaluation on a dataset constructed from Defects4J open-source projects [12]. Scate improves line and branch coverage over unsupervised coding agent baselines by 32.3% and 30.9%, respectively, with Gemini-cli, and by 6.0% and 5.9%, respectively, with Claude Code, demonstrating its effectiveness across diverse coding agents. It also consistently outperforms state-of-the-art non-agentic approaches across all evaluation metrics.

2 Approach↩︎

To fully automate unit test generation, Scate employs a supervisor-orchestrated agentic workflow. The supervisor first assesses the target class using structural testability metrics. It then combines these static characteristics with dynamic feedback from the evolving test suite, including achieved coverage and uncovered complexity, to assess the current testing context and select the action most likely to improve test generation outcomes. At each timestep, the supervisor adapts its policy to balance coverage gains against computational cost, deciding whether to continue standard generation, invoke deeper program analysis for complex branches, or terminate the process to avoid resource exhaustion. 2 provides an overview of the workflow. The following subsections describe the supervisor formulation, program analysis mechanisms, and supervisor-agent interaction.

2.1 Contextual Bandit Formulation↩︎

We formulate the Scate supervisor as a contextual bandit problem [13]. As a mature branch of reinforcement learning (RL) [14], the contextual bandit framework is highly effective at selecting optimal actions based on rich state observations [15], aligning directly with the context-aware design of Scate. Unlike full RL formulations, which require modelling complex, sequential state transitions over thousands of training episodes, a contextual bandit efficiently learns a direct policy by optimizing for immediate, single-step payoffs. To formalize this approach, we first define the contextual features that capture the static and dynamic state of the target class, followed by the supervisor’s action space. We then present the reward function used to balance coverage gains against computational cost and conclude with the learning algorithm that derives the decision-making policy.

2.1.1 Contextual Features↩︎

Generating comprehensive unit tests requires LLM-based coding agents to understand source code, including complex control flows and extensive external dependencies, within a limited context window [2], [3]. The inherent complexity of a target class directly influences both achievable code coverage and the effort required to improve it. More complex classes are typically harder to test, achieve lower coverage, and require substantially greater agent effort.

We leverage class testability metrics to quantify the computational and reasoning demands of unit test generation [16]–[19]. Although originally proposed to estimate human testing effort [20], these metrics can serve as proxies for the challenges faced by coding agents. For example, high internal complexity increases the reasoning burden on the LLM, while high external coupling complicates environment setup and dependency mocking.

To construct a compact and informative state representation, we draw on the testability metrics identified by Bruntink et al. [17] and select a small set of aggregate, non-redundant features. Specifically, our state representation comprises three class-level metrics that characterize overall size, internal complexity, and external coupling:

• Lines of Code (LOC): It acts as a baseline measure of class size, serving as a proxy for the foundational context window required by the LLM to parse the target class.

• Weighted Methods per Class (WMC): This metric represents the sum of the complexities of the methods within a given class, where each method is weighted by its cyclomatic complexity [21]. It serves as a strong indicator of internal structural difficulty and the risk of agent reasoning failures.

• Response for a Class (RFC): Defined as the number of methods within the targeted class and the number of methods from other classes it invokes. This metric serves as a proxy for external dependencies and the complexity of the mocking required by the agent.

These metrics are computed using the control flow graph (CFG) of the target class. While static metrics characterize the inherent difficulty of the target class, the supervisor must also track generation progress. We therefore incorporate real-time line coverage (\(Cov^{L}\)), branch coverage (\(Cov^{B}\)), and missed_complexity, defined as the number of uncovered method entries and decision points [22]. Unlike relative coverage percentages, which can obscure the true amount of remaining work, missed_complexity provides a direct measure of the unresolved structural testing opportunity. Thus, our resulting contextual feature vector comprises three static metrics (LOC, WMC, and RFC) and three dynamic metrics (line coverage, branch coverage, and missed_complexity).

In practice, software structural metrics typically follow a power-law distribution. To ensure numerical stability and prevent classes with extreme complexity from disproportionately skewing the downstream learning process, we apply logarithmic scaling [23] to compress and normalize unbounded metrics as \(\tilde{\text{{\small\ttfamily\texttt{metric}}}} = \ln(1 + \text{{\small\ttfamily\texttt{metric}}})/10\). At any discrete time step \(t\), the complete contextual state observed by the supervisor is formally defined as the vector:

\[\label{eq:context-vector} \small \mathbf{x}_t = \big[ 1, \tilde{L}, \tilde{W}, \tilde{R}, Cov^{L}, Cov^{B}, \tilde{M} \big]^\top\tag{1}\]

Where:

• \(1\) acts as an intercept, ensuring baseline stability for the downstream decision policy.

• \(\tilde{L}, \tilde{W}, \tilde{R}\) and \(\tilde{M}\) represent the normalized LOC, WMC, RFC and missed_complexity metrics, respectively.

• \(Cov^L, Cov^B \in [0,1]\) represent the current line and branch coverage.

2.1.2 Action Space↩︎

Following the supervision challenge outlined in Section 1, we model agent guidance as a routing problem with three possible actions: Default, Analysis, and Stop. Constraining the action space to exactly these three macro-level choices is a deliberate mathematical and architectural design decision. A broader action space (e.g., forcing the supervisor to micro-manage individual analysis tools or prompt variations) would drastically expand the exploration space, slowing the contextual bandit’s convergence and hindering its ability to efficiently discover coverage-maximizing strategies. Instead, these three arms encapsulate the entire intervention spectrum (baseline generation, targeted assistance, and zero-cost termination), enabling the supervisor to rapidly learn a cost-effective policy that maximizes coverage. We define the discrete action space as \(\mathcal{A} = \{a_{\text{def}}, a_{\text{ana}}, a_{\text{stop}}\}\), representing the three available routing decisions:

1. Default (\(a_{\text{def}}\)): A direct, zero-shot generation strategy. Relying solely on predefined project context.

2. Analysis (\(a_{\text{ana}}\)): A tool-augmented generation strategy. The agent dynamically queries the Scate MCP tool, our custom program analysis tool deployed as a Model Context Protocol (MCP)  [24] server, to navigate uncovered paths, resolve complex logical constraints, and identify mocking targets for complex code.

3. Stop (\(a_{\text{stop}}\)): A termination signal to halt execution and save computational resources.

Thus, while the Default action provides a rapid baseline for easily covered structures, the supervisor learns to dynamically allocate the tool-augmented Analysis action to complex classes where zero-shot generation reaches a plateau in capability. Ultimately, it chooses Stop to prevent resource exhaustion when further generation attempts yield diminishing returns.

2.1.3 Reward Function↩︎

Over a sequence of discrete timesteps \(t = 1, 2, \dots, T\), where \(T\) represents the total decision-making steps made by the supervisor, the system iteratively refines its policy. At each step \(t\), after the supervisor evaluates the current contextual state vector \(\mathbf{x}_t\) and selects an action \(a_t \in \mathcal{A}\), the coding agent executes the corresponding generation strategy. The environment then returns an action-dependent reward signal \(r_t\), providing critical feedback to update the supervisor’s underlying policy. To achieve our objective of maximizing overall code coverage while minimizing cost (i.e., token usage), we design a reward function that formalizes this trade-off for every discrete action.

Consistent with optimal stopping theory [25], we define the reward for the Stop action as zero (\(r_t = 0\) when \(a_t = a_{\text{stop}}\)), reflecting a terminal state with neither additional cost nor benefit. Conversely, the reward signal for active decisions (Default and Analysis) is designed to capture both the computational cost and the coverage gain incurred at step \(t\). We calculate the relative coverage gain \(g_t\), which prevents early-round bias by measuring the coverage improvement strictly relative to the remaining uncovered code. Let \(Cov_{t-1}^{L}, Cov_{t-1}^{B} \in [0, 1]\) denote the initial line and branch coverage at the start of step \(t\), and \(Cov_{t}^{L}, Cov_{t}^{B} \in [0, 1]\) denote the coverage after the action is executed. The relative gain is defined as: \[\label{eq:relative-cov} \small g_t = \frac{1}{2} \left( \frac{Cov_t^L - Cov_{t-1}^L}{\max(1 - Cov_{t-1}^L, \epsilon)} + \frac{Cov_t^B - Cov_{t-1}^B}{\max(1 - Cov_{t-1}^B, \epsilon)} \right)\tag{2}\]

where \(\epsilon = 10^{-5}\) is a small constant ensuring numerical stability when coverage nears \(1.0\). This formulation correctly values late-stage, complex test generation higher than raw absolute gains. For example, an absolute coverage increase of \(0.05\) (from \(0.90\) to \(0.95\)) yields a relative gain of \(0.50\), representing the resolution of half the remaining uncovered code. By assigning greater rewards to late-stage coverage improvements, the formulation incentivizes the supervisor to pursue additional gains, such as transitioning from Default generation to Analysis when coverage growth begins to plateau.

As discussed in Section 2.1.1, achieving a given coverage level is substantially more difficult for complex classes than for simpler ones. To account for this disparity, we scale the relative coverage gain \(g_t\) using a complexity multiplier, \(m_t = 1 + \ln(1 + c_t)\), where \(c_t = \max(0, Comp_t - Comp_{t-1})\) denotes the increase in covered complexity during step \(t\), corresponding to the reduction in missed_complexity. This formulation rewards progress on structurally complex classes while the logarithmic transformation preserves stable bandit updates.

To account for cost, each generation action incurs an API token cost, \(Cost_t\), defined as the total monetary expense (in USD) associated with all input, output, and cached tokens extracted from the coding agent’s log files. By default, \(\omega = 1\), balancing complexity-scaled coverage gains and cost.

Finally, to discourage actions that do not improve coverage (\(g_t \le 0\)), we apply a fixed failure penalty (\(\kappa = 0.5\)). This value was empirically chosen to balance discouraging ineffective actions with maintaining exploration. The resulting action-dependent reward \(r_t\) is defined as:

\[\label{eq:reward} \small r_t = \begin{cases} 0 & \text{if } a_t = a_{\text{stop}} \\ g_t \cdot m_t - \omega \cdot Cost_t & \text{if } a_t \neq a_{\text{stop}} \text{ and } g_t > 0 \\ - \omega \cdot Cost_t - \kappa & \text{if } a_t \neq a_{\text{stop}} \text{ and } g_t \le 0 \end{cases}\tag{3}\]

2.1.4 Contextual Bandit Algorithm↩︎

With the contextual state vector \(\mathbf{x}_t\), action space \(\mathcal{A}\), and reward function \(r_t\) established, we operationalize the supervisor’s decision-making policy using the LinUCB algorithm [13]. LinUCB is a highly efficient contextual bandit algorithm designed to dynamically balance the exploration of novel strategies with the exploitation of known successes. The core components of LinUCB are detailed in Algorithm 3.

The central optimization objective over the global horizon \(T\) is to maximize the expected cumulative reward, denoted formally as \(\mathbb{E} \big[ \sum_{t=1}^T r_t \big]\). To achieve this, the supervisor must consistently approximate the optimal action \(a_t^*\) that yields the maximum expected reward at each discrete trial \(t\). LinUCB facilitates this optimization by operating under the assumption [13] that the expected reward for any action \(a\) is a linear function of the context \(\mathbf{x}_t\), parameterized by an unknown weight vector \(\boldsymbol{\theta}^*_a\): \(\mathbb{E}[r_{t}|a, \mathbf{x}_t] = \mathbf{x}_t^\top \boldsymbol{\theta}^*_a\). To learn these hidden parameters, the algorithm maintains a covariance matrix \(\mathbf{A}_a \in \mathbb{R}^{d \times d}\) (where \(d=7\) as defined in Section 2.1.1) and a cumulative reward vector \(\mathbf{b}_a \in \mathbb{R}^d\) for each action \(a \in \mathcal{A}\). To ensure continuity across all timesteps, these structures (\(\mathbf{A}\) and \(\mathbf{b}\)) are either loaded from persistent storage or, if no prior state exists, initialized as an identity matrix (\(\mathbf{I}_d\)) and a zero vector (\(\mathbf{0}_d\)) to provide an unbiased baseline (Lines [line:initial-start]–[line:initial-end]). During this iterative refinement, these matrices act as sufficient statistics: \(\mathbf{A}_a\) records the outer products of past contexts (\(\mathbf{x}_t \mathbf{x}_t^\top\)) at Line [line:update95A], while \(\mathbf{b}_a\) aggregates the context-weighted rewards (\(r_t \mathbf{x}_t\)) at Line [line:update95b]. This state is saved to path \(\mathcal{P}\) after each update to ensure persistence.

To govern action selection at timestep \(t\), the supervisor evaluates the current context \(\mathbf{x}_t\) and computes the Upper Confidence Bound [14] (UCB) score \(p_{t,a}\) for each available strategy. Because the reward for the Stop action is a deterministic baseline (\(r_t = 0\)), its expected score carries no uncertainty and is strictly evaluated as \(0.0\) (Line [line:stop]). For the active generation strategies, the supervisor first derives the current empirical weight vector by solving the ridge regression estimate \(\hat{\boldsymbol{\theta}}_a = \mathbf{A}_a^{-1}\mathbf{b}_a\) (Line [line:theta]). It then calculates \(p_{t,a}\) by combining this exploitative estimate (\(\mathbf{x}_t^\top \hat{\boldsymbol{\theta}}_a\)) with a mathematical exploration term (\(\alpha \sqrt{\mathbf{x}_t^\top \mathbf{A}_a^{-1} \mathbf{x}_t}\)), which acts as an uncertainty bonus scaled by exploration hyperparameter \(\alpha\) (Line [line:ucb95calc]). As the supervisor gathers more observations, the continuous refinement of \(\mathbf{A}_a\) and \(\mathbf{b}_a\) drives the empirical estimate \(\hat{\boldsymbol{\theta}}_a\) closer to the true optimal parameters \(\boldsymbol{\theta}^*_a\). Concurrently, the uncertainty bound geometrically shrinks, naturally shifting the supervisor’s policy from broad exploration to confident exploitation. The final action \(a_t\) with the highest overall UCB score is then selected (Line [line:argmax]):

\[\label{eq:action} \small a_t = \operatorname*{argmax}_{a \in \mathcal{A}} (p_{t,a})\tag{4}\]

2.2 Prompt Design and Tool Integration↩︎

Following the LinUCB supervisor’s selection of an action—either Default or Analysis—the framework dynamically generates a context-aware prompt to guide the coding agent’s test generation process.

2.2.1 Default Prompt↩︎

If the supervisor selects the Default action (\(a_{\text{def}}\)), the framework injects the baseline prompt template shown in Fig. 4. This template serves as the foundational system instruction, providing the coding agent with a strictly defined operational boundary and execution strategy. The prompt guides the coding agent to operate as an autonomous developer. It dynamically anchors the agent to the target class, establishes rigid success criteria, and supplies the necessary commands for environmental awareness. Crucially, rather than relying on a single attempt to generate tests, it enforces an iterative workflow, instructing the agent to actively identify gaps, synthesize tests, and independently repair failures based on execution feedback until code coverage is maximized.

None

2.2.2 Analysis Prompt and Scate MCP Integration↩︎

When the LinUCB supervisor determines that a target class requires deeper structural reasoning, it selects the Analysis action (\(a_{\text{ana}}\)). In this state, our Scate framework performs two critical operations to enable the Analysis action. First, it launches a program analysis tool get_class_analysis as an MCP server, making its program analysis capabilities seamlessly accessible to the coding agent. Second, it dynamically augments the Default prompt by appending the explicit MCP tool instructions (see the Analysis Prompt section in 4. The complete Analysis prompt template is included in our replication package. Whereas the Default prompt relies entirely on the agent’s intrinsic reasoning over coverage reports, the Analysis prompt enhances its program analysis capabilities through integration with the Scate MCP tool.

For the implementation of get_class_analysis, we leverage the CFG as a middleware alongside coverage reports to extract uncovered execution paths. A significant challenge in this process is information overload: a highly complex class with low code coverage may contain hundreds of uncovered paths. Passing all of these paths simultaneously would overwhelm the context window of the underlying LLM and degrade the agent’s focus. Conversely, providing only a single path per prompt would require excessive, high-latency LLM queries, wasting the model’s reasoning capabilities.

To balance this tradeoff, we design the tool to deliver paths in batches based on empirical observations. After extracting all paths containing uncovered lines and branches, Scate MCP sorts the methods by their number of uncovered statements. Within each method, the uncovered paths are subsequently sorted to identify the most critical path, defined as the path containing the highest number of uncovered statements, with any ties broken by selecting the shortest path. A single tool invocation provides the coding agent with a maximum of 10 under-covered methods, each accompanied by its most critical uncovered path. This targeted strategy allows the coding agent to maintain strict focus on generating tests for specific, high-priority coverage gaps.

Beyond execution paths, Scate MCP utilizes data flow analysis to resolve external method calls and their associated class types for each method under test. The coding agent can leverage this data to configure required mock objects or test setups, and to deduce the underlying behavior of the method by observing its external interactions. The extraction of uncovered paths and external dependencies directly addresses the testability bottlenecks quantified by the WMC and RFC metrics, which are integrated into the supervisor’s contextual feature vector (Section 2.1.1). An example of the resulting JSON payload is shown in 5, illustrating the analysis of an under-covered handleConcatenatedOptions method from the DefaultParser class.

None

2.3 Supervised Test Generation Workflow↩︎

Algorithm 6 illustrates our workflow for a single target class, where the contextual bandit supervisor dictates the high-level strategy, while the agent executes the test generation.

The process begins by initializing the LinUCB policy (\(\pi\)), the core decision-making engine of the supervisor, by loading existing policy weights from the storage path (\(\mathcal{P}\)) via the Initialize function (Line [line:init-policy]). Next, Scate extracts the class metrics for the static portion of the context vector (Line [line:init-metrics]).

We define the test generation process for a single target class as a generation trajectory. To prevent infinite loops and optimize computational resources, the framework restricts each trajectory to a maximum number of iterations, \(K_{max}\) (Line [line:max-interation]). Note that while \(k < K_{max}\) indexes the iterations within a single trajectory, the supervisor tracks its continuous adaptation using a global timestep, \(t\), which accumulates all actions taken across all previous classes. At the start of each iteration \(k\), Scate evaluates the current line and branch coverage. Simultaneously, it extracts and stores the existing valid test code (\(TestCode_{valid}\)) to establish a safe baseline for potential rollbacks (Lines [line:init-cov]–[line:init-testcode]). If both line and branch coverage have already reached the threshold of \(100\%\), further generation is unnecessary, and the trajectory terminates early without requiring the supervisor’s intervention (Lines [line:saturated-start]–[line:saturated-end]).

If the trajectory continues, the framework dynamically constructs the context vector (\(\mathbf{x}_k\)) by fusing the static structural metrics with the current coverage status (Line [line:context-vector], detailed in Equation 1 ). The supervisor uses this context to select the current optimal action \(a_k\) (Line [line:select-action]). If the policy selects the \(a_\text{stop}\) action, it signals that further generation efforts are likely futile; the process terminates, and the policy is updated with a reward of 0 (Lines [line:stop-start]–[line:stop-end]).

If the supervisor selects an active generation action (Default or Analysis), Scate prepares the corresponding prompt and environment settings, instructing the agent to generate new tests (Lines [line:prompt]–[line:agent-run]). Upon completion, the framework records the token cost (\(Cost_k\)) and strictly evaluates the newly generated test suite. Because LLMs are prone to hallucinations that can introduce syntax errors or regress coverage, Scate incorporates a guardrail mechanism. A new test suite is only accepted if it successfully compiles, passes all assertions, and does not regress existing coverage (\(NoRegression\)) (Lines [line:evaluate]–[line:accept]). If any of these conditions fail, the rollback mechanism discards the corrupted generation and restores the file to the previously saved \(TestCode_{valid}\) state (Lines [line:rollback-start]–[line:rollback-end]).

Finally, the framework calculates the reward signal (\(r_k\)) based on the net coverage improvement and the incurred token cost (Line [line:reward]). This feedback is used to adapt the LinUCB policy weights for the selected action (Line [line:adapt-policy]). This cycle repeats until the test suite is fully saturated, the supervisor selects the stop action, or the iteration limit \(K_{max}\) is reached, at which point the final optimized test suite is returned.

Scate operates in an online learning setting. Rather than learning from project-specific source code, the supervisor learns exclusively from contextual features derived from class-level testability metrics and generation feedback. After each generation action, the observed reward is used to update the LinUCB policy, thereby enabling the supervisor to continuously refine its decision-making strategy. To facilitate initial policy calibration, the supervisor is first exposed to a brief exploration period before transitioning to exploitation-guided decision making. During deployment, policy updates continue throughout test generation, allowing the supervisor to adapt to previously unseen classes. The specific training and evaluation used in our experiments are described in the next section.

3 Evaluation↩︎

Our study addresses the following research questions:

• RQ1: How does Scate’s adaptive policy evolve during execution, and how does it respond to varying contextual characteristics?

• RQ2: What is the impact of Scate’s adaptive supervision on the effectiveness of coding agents?

• RQ3: How effective is Scate compared to state-of-the-art non-agentic LLM-based approaches?

3.1 Experimental Setup↩︎

3.1.1 Dataset↩︎

To evaluate Scate, we selected the same Defects4J projects [12] used in the evaluation of the state-of-the-art LLM-based test generation approach, Panta [3]. Unlike Panta, which relies solely on cyclomatic complexity, Scate utilizes a set of class metrics (LOC, WMC, RFC). Since the primary objective of Scate is to supervise coding agents on challenging test-generation tasks, we focus our evaluation on structurally complex classes. Following established thresholds [26], we retained classes with WMC \(\geq 35\) or RFC \(\geq 55\), and selected up to 10 qualifying classes per project to ensure project diversity while limiting the computational cost of repeated experimental runs. This process yielded a final dataset of 120 complex classes across 14 projects. The resulting dataset exhibits substantial structural complexity, with median values of 416.5 LOC, 68.0 WMC, and 80.5 RFC. Complete project-level metric distributions are provided in our replication package.

3.1.2 Online Learning↩︎

Because Scate employs a contextual bandit design for continuous adaptation, we evaluate it using an online learning setup. Since the supervisor does not learn from project-specific source code, all 120 classes were randomly shuffled, regardless of project origin, and partitioned into a 40-class training set and an 80-class evaluation set. As shown in Figure 7, the two partitions exhibit comparable complexity distributions (median WMC of 67.5 and 70.0, respectively). Complete distribution statistics are available in our replication package. The training phase consists of two stages. First, a random warm-up phase uses the first 20 training classes for pure exploration. For each class, the supervisor selects generation actions uniformly at random for up to \(K_{max}=10\) iterations, allowing unbiased collection of contextual and reward information. Second, an \(\alpha\)-decay phase uses the remaining 20 training classes to transition from exploration to exploitation. During this stage, the supervisor selects actions using LinUCB while gradually decaying the exploration parameter from \(\alpha=1.5\) to \(\alpha=0.2\) [27]–[29]. Following training, we evaluate Scate on the remaining 80 unseen classes. During evaluation, the supervisor continues to update its policy online while maintaining a fixed exploration parameter of \(\alpha=0.2\), simulating a deployment scenario in which supervision quality improves through continued interaction with new classes.

3.1.3 Agents↩︎

To address RQ1—RQ3, we employ Gemini-cli [9] (v0.35.3) as the primary coding agent within Scate, configured with the gemini-3.1-flash-lite model (temperature = 1.0). We selected this model due to its support for tool calling, relatively low inference cost [30], and stable availability [31] throughout the experimental period. For RQ2, we also evaluate Scate with Claude Code [8] (v2.1.173) using claude-4-5-haiku, a model selected for its comparable speed and cost profile.

Scate supports both augmenting existing test suites and generating tests from scratch. To ensure a fair comparison across all research questions, we evaluate the latter scenario, where no existing test cases are provided and coverage for each target class starts at zero. Our experiments were conducted between April–June 2026.

3.2 RQ1: Execution Dynamics↩︎

To address RQ1, we investigate the execution dynamics of Scate’s policy to understand what the supervisor learns and how its behavior evolves.

3.2.1 Learning Progress and Policy Adaptation↩︎

To evaluate the learning efficacy, we analyzed the lifetime average cumulative reward \(\bar{R}_t = \frac{1}{t} \sum_{i=1}^{t} r_i\) across all timesteps \(t \in T\). This step-wise formulation provides a high-fidelity representation of the policy’s learning progress over time  [32].

As shown in Figure 8, the average cumulative reward remains near 0.2 during the random warm-up phase (Classes 1–20 | Timesteps 0–199), where actions are selected primarily for exploration. During the subsequent \(\alpha\)-decay phase (Classes 21–40 | Timesteps 200–378), the supervisor progressively shifts toward exploitation, resulting in a steady increase in average cumulative reward to 0.3. This improvement suggests that the supervisor has learned to make more effective routing decisions based on the observed testing context.

Following training, the exploration rate stabilizes at \(\alpha = 0.2\), prioritizing exploitation while maintaining online adaptation. During evaluation (Classes 41–120), the average cumulative reward remains stable despite a brief decline on several highly complex classes and subsequently resumes its upward trend. This result suggests that the supervisor continues to improve its decision-making over time.

3.2.2 Intra-Trajectory Analysis of Supervisor Behavior↩︎

Following the assessment of overall learning progress, we analyze the behavior of the learned policy on the 80 evaluation classes. Figure 9 details the execution dynamics within trajectories across the 10-iteration generation horizon.

The largest coverage gains occur early in the trajectory. Iteration 1 yields an average coverage increase of 43.8%, followed by an additional 11.9% in Iteration 2. During this phase, the supervisor relies almost exclusively on the Default action, with only six classes invoking Analysis in Iteration 2. This behavior suggests that when generation begins from zero coverage, the standard Default prompt is generally sufficient to rapidly capture easily reachable coverage.

As the trajectory progresses, the rate of coverage improvement naturally decreases as the remaining uncovered paths become increasingly complex. Despite that, the supervisor continues to achieve positive coverage gains throughout the entire 10-iteration horizon. It consistently allocates a subset of classes to Analysis, with 11–15 classes invoking deeper program analysis per iteration between Iterations 4–10. This behavior suggests that the learned policy reserves Analysis for classes where additional coverage remains achievable despite increasing structural complexity and diminishing returns from standard generation.

As returns diminish, the supervisor increasingly issues Stop actions when neither Default nor Analysis yields sufficient reward. The cumulative number of early-terminated classes steadily increases, ultimately reaching 32 of the 80 evaluation classes (40%) by Iteration 10. This aligns with the reward formulation described in Section 2.1.3, which balances coverage gains against computational cost. Because the API cost of gemini-3.1-flash-lite remains low (below $0.10 USD per iteration), the cost component contributes only a modest penalty. Consequently, the learned policy prioritizes extracting additional coverage when meaningful gains remain available while avoiding premature termination.

By Iteration 10, the average coverage gain drops to 0.8%, while 40% of classes have already been terminated early. Together, these observations suggest that the predefined \(K_{max}=10\) cutoff effectively bounds generation cost without sacrificing substantial additional coverage. Overall, the average trajectory length is 7.9 iterations. By dynamically orchestrating different actions throughout the generation process, Scate demonstrates its ability to adapt action selection to the contextual characteristics of the target class, balancing coverage improvement against computational cost.

3.2.3 Impact of Contextual Features on Action Selection↩︎

To further understand the supervisor’s decision-making mechanism, we analyze how the contextual state of the target class influences the Scate supervisor’s action selection. In our formulation, context comprises both static class metrics and dynamic code coverage. To structure this analysis, we categorize the generation trajectories by the types of actions selected and the total length of the trajectory before termination (\(\le 5\) vs.\(> 5\)). Specifically, Default only refers to generation trajectories where the supervisor relies exclusively on the Default action across all iterations. Conversely, with Analysis denotes trajectories where the supervisor invokes Analysis action in at least one iteration before termination. Table ¿tbl:tab:bandit95context? presents the average values of these contextual features across the classes in each category.

As shown in the upper half of Table ¿tbl:tab:bandit95context?, the static complexity of a target class heavily influences the supervisor’s decision to invoke the Analysis action. Trajectories categorized as Default only are associated with smaller, simpler classes (average LOC of 469.3, WMC of 98.5, and RFC of 90.3) that yield high average coverage (\(Cov^L = 85.9\%\), \(Cov^B = 78.7\%\)). For these classes, the supervisor correctly learns that the Default prompt is sufficient to achieve adequate code coverage. By contrast, the supervisor learns to deploy the Analysis action for significantly larger and more complex classes (average LOC of 755.1, WMC of 163.9, and RFC of 104.8). Due to this inherent complexity, the final achieved code coverage for these classes is lower (76.6% line and 66.7% branch coverage). This indicates that the supervisor correctly determines when the Analysis action is required, i.e., when a target class possesses complex logic, and the standard Default mode is no longer sufficient to increase coverage.

The bottom half of Table ¿tbl:tab:bandit95context? reveals how contextual state dictates the trajectory length and the timing of the Stop action. Short trajectories (\(\le 5\) iterations) are strongly associated with simpler classes (average LOC of 443.0, WMC of 83.2, and RFC of 87.3), where high code coverage (90.0% line and 85.8% branch coverage) is easily achieved in fewer iterations. Once coverage saturates on these simpler classes, the supervisor executes an early Stop to prevent wasteful iterations that yield no further reward. In contrast, longer trajectories (\(> 5\) iterations) correlate with higher static complexity (average LOC of 601.4, WMC of 131.4, and RFC of 97.5) and lower overall coverage levels (80.4% line and 71.4% branch coverage). For these challenging classes, the supervisor persists longer, maintaining active generation and analysis to incrementally extract remaining coverage gains until diminishing returns force termination.

3.3 RQ2: Supervised vs. Unsupervised Agents↩︎

We compare against four standalone-agent baselines that operate without the supervisor. The first two baselines represent single-shot executions: Agent (Default), which utilizes the standard Default prompt, and Agent (Analysis), which utilizes the Analysis prompt alongside the Scate MCP tool. Finally, to ensure a rigorously fair comparison against Scate’s multi-iteration trajectories, with an average of 7.9 iterations across the evaluation set, we introduce two iterative baselines: Agent (Default, 8 Iters) and Agent (Analysis, 8 Iters). These configurations force the agent to generate tests for eight consecutive iterations using their respective setups. The baselines and Scate use Gemini-cli as the underlying coding agent. All prompts are included in our replication package.

Table 1 reports the per-class average values across the evaluation set for line coverage (\(Cov^L\)), branch coverage (\(Cov^B\)), mutation score (mutation %), cost, generation iterations, and overall execution time. As shown, the single-shot baselines demonstrate that deep analysis is not strictly advantageous for initial test generation. Agent (Default) achieves comparable or slightly superior coverage (\(Cov^L\) of 50.5%, \(Cov^B\) of 43.8%) compared to Agent (Analysis) (\(Cov^L\) of 49.6%, \(Cov^B\) of 43.9%), while operating at a lower cost ($0.079 vs. $0.085). This validates Scate’s learned behavior observed in RQ1, where the supervisor universally selected the Default action for the first iteration to efficiently extract easily achievable coverage. Furthermore, these results highlight the inherent limitations of single-shot generation: restricted to a single iteration, both baselines fail to achieve high overall code coverage, falling nearly 31% behind Scate in branch coverage.

1.5pt

The Agent (Analysis, 8 Iters) baseline achieves a line coverage of 74.7% and a branch coverage of 67.5%. We use a one-sided Wilcoxon signed-rank test (\(\alpha = 0.05\)) to evaluate differences between Scate and the baseline, as it is a robust nonparametric method well-suited to paired, non-normally distributed software metrics. This test confirms that Scate yields statistically significant improvements, achieving 82.8% line coverage (+8.1%, \(p < 0.001\)) and 74.7% branch coverage (+7.2%, \(p < 0.001\)) within an equivalent average iteration footprint. Similarly, when compared with the Agent (Default, 8 Iters) baseline, Scate achieves 5.5% higher line coverage (\(p = 0.007\)) and 4.5% higher branch coverage (\(p = 0.004\)).

In addition to line and branch coverage, we measure mutation score using Pitest (v1.17.0). Scate achieves an average mutation score of 64.7%, significantly outperforming both the Agent (Analysis, 8 Iters) (58.1%, \(p<0.001\)) and Agent (Default, 8 Iters) (59.6%, \(p=0.002\)) baselines. In terms of both cost and runtime, Scate falls between the two baselines. It averages $0.693 and 15.1 minutes, which is slightly higher than Agent (Analysis, 8 Iters) ($0.567, 14.5 min) but lower than Agent (Default, 8 Iters) ($0.797, 16.5 min). These results indicate that Scate’s significant gains in coverage and mutation scores come without substantial computational overhead. Ultimately, the comparison with both iterative baselines demonstrates that the supervisor’s ability to dynamically interleave actions based on real-time structural context outperforms a static, homogeneous agent loop.

Additionally, to assess the impact of the underlying coding agent, we compare Scate when paired with Claude Code and Gemini-cli. Unsupervised, Claude Code achieves substantially higher initial coverage (84.3% line and 79.0% branch) than Gemini-cli(50.5% and 43.8%, respectively), but at a considerably higher Iteration 1 cost ($0.518 vs. $0.079). Consequently, Scate adapts its supervision policy by terminating Claude Code early for 95.0% of classes, resulting in short trajectories averaging 2.1 iterations and only 10.0% of classes invoking Analysis. In contrast, Gemini-cli’s lower cost and greater opportunity for coverage improvement lead to longer trajectories (7.9 iterations on average) and more frequent Analysis usage (33.8%). Ultimately, Scate paired with Claude Code achieves higher final coverage (90.3% line and 84.9% branch) alongside a 75.0% mutation score at a total cost of $1.143, while with Gemini-cli it reaches competitive coverage (82.8% line and 74.7% branch) and a 64.7% mutation score for $0.693. These results demonstrate that Scate dynamically adapts its supervision strategy to the capabilities and cost profile of the underlying coding agent.

3.4 RQ3: Non-Agentic State-of-the-Art Comparison↩︎

We compare Scate against two state-of-the-art LLM-based test generation approaches, Panta [3] and SymPrompt [2], which combine program analysis with prompt-based strategies. While Panta is publicly available [33], SymPrompt is not; therefore, we use the high-fidelity SymPrompt replication provided by the Panta authors [33]. To ensure a fair comparison, all approaches are evaluated on the same 80-class dataset using the identical underlying model (gemini-3.1-flash-lite, temperature = 1.0). This isolates the impact of Scate’s supervised agentic workflow relative to static prompting-based generation pipelines.

Figure 10 shows the distributions of coverage and mutation score using violin plots with embedded boxplots. Statistical significance is assessed using a one-sided Wilcoxon signed-rank test (\(\alpha = 0.05\)). As illustrated, Scate consistently outperforms both Panta and SymPrompt across all three metrics. For Line Coverage, Scate achieves a mean of 82.8%, surpassing Panta(77.2%, \(p=0.007\)) and SymPrompt(44.7%, \(p<0.001\)). Beyond these higher averages, Scate’s top-heavy distribution indicates a high concentration of classes achieving near-perfect coverage. Furthermore, Scate establishes a substantially higher empirical minimum compared to both baselines. In contrast, SymPrompt exhibits a severely bottom-heavy distribution extending down to 0%, reflecting a high frequency of low-coverage outcomes. Moving to Branch Coverage, the data reflects a similar structural advantage: Scate maintains a superior mean of 74.7% and a concentrated upper quartile, whereas Panta(70.2%, \(p=0.009\)) and SymPrompt(37.4%, \(p<0.001\)) exhibit pronounced lower tails, demonstrating that static prompting frequently fails to navigate intricate path logic, while Scate successfully resolves these complex conditions to achieve high branch coverage.

Most critically, the Mutation Score validates Scate’s effectiveness beyond mere structural execution by evaluating the suite’s fault-detection capability. Scate achieves a leading mean of 64.7% and maintains a robust empirical floor consistently near or above the 20% threshold. Conversely, both Panta(61.6%, \(p=0.017\)) and SymPrompt(33.7%, \(p<0.001\)) exhibit a high density of classes falling below 20%. Ultimately, these results demonstrate Scate’s superior capability in generating high-quality test suites that maximize both code coverage and overall test effectiveness.

4 Discussion↩︎

Application. Because the supervisor relies entirely on source-code-independent numerical metrics rather than specific syntax, the learned policy generalizes well beyond the Defects4J dataset. To support out-of-the-box usage, our replication package includes pre-trained policies for Gemini-cli and Claude Code. Users can also train Scate for other coding agents (e.g., Qwen Code) or different underlying LLMs.

Efficiency and Cost. Training a customized Scate policy is a one-time investment. For our 40-class training set, policy learning required 13 hours and $48.40 using Gemini-cli, compared with 22.7 hours and $127.00 using Claude Code. Once trained, generating a test suite required an average of 15.1 minutes and $0.69 per class with Gemini-cli, and 11.1 minutes and $1.14 with Claude Code. Generation time scaled with class complexity (1–49 minutes for Gemini-cli and 2–60 minutes for Claude Code). Because Scate integrates directly with standard coding agents, users with commercial subscriptions can typically absorb both the training and generation cost through their existing usage quotas.

Agentic vs. Non-Agentic. Agentic and non-agentic test generation represent different design philosophies. Non-agentic approaches, such as Panta and SymPrompt, operate on narrow contexts using carefully engineered prompting pipelines, resulting in lower inference cost but limited reasoning capability. In contrast, agentic approaches leverage repository-wide context, iterative planning, tool invocation, and multi-step reasoning, enabling them to address complex dependencies and substantially improve test quality at the expense of additional computation. Our results demonstrate that this tradeoff yields significantly higher code coverage and mutation scores despite the increased inference cost.

Threats To Validity. A threat to internal validity is the stochasticity of both the coding agent and the contextual bandit. We mitigate this by normalizing rewards and contextual features, randomly shuffling the training and evaluation classes, evaluating only after empirical policy convergence, and reporting aggregate results with statistical significance testing. Together, these measures reduce the influence of stochastic variation and class-ordering bias.

External validity may be limited by our use of a single benchmark (Defects4J) and programming language (Java). However, both the supervisor formulation and the underlying coding agents are language-agnostic. Because Defects4J projects are publicly available, LLM pretraining contamination is also a potential concern. To ensure fair comparisons, all evaluated approaches use the same underlying models and configurations, and we emphasize relative improvements over baselines rather than absolute performance. Finally, although training and evaluation classes originate from the same projects, data leakage is mitigated because the supervisor learns exclusively from project-independent numerical features rather than source code.

Reproducibility. Code and datasets will be made publicly available upon publication.

5 Related Work↩︎

Automated unit test generation has been a longstanding research topic, with approaches historically falling into two main categories: search-based  [34]–[37] and learning-based  [38]–[40]. The rapid evolution of Large Language Models (LLMs) has led to their widespread adoption for tasks related to automated test generation  [5], [6], [41]–[46]. Empirical studies have also been conducted to evaluate the test generation capabilities of different LLMs in various programming languages  [47]–[53]. To tackle complex code, static tools such as HITS [1], JUnitGenie [54], and SymPrompt [2] rely on program analysis, prompting LLMs with context extracted as method slices or execution paths, whereas CoverUp [4] overcomes their inability to augment existing test suites by leveraging dynamic runtime coverage. Panta [3] combines dynamic coverage with static control-flow analysis to generate new tests and augment existing ones. While other techniques primarily focus on improving code coverage, MutGen [55] shifts its objective toward maximizing the mutation score.

Contextual bandits [13] are a specialized form of reinforcement learning designed to map contexts to actions that maximize an immediate reward. Within the broader AI community, this approach has recently guided dynamic LLM routing in tools such as MetaLLM [29], LLMBandit [56], and MixLLM [57]. Concurrently, general AI frameworks such as SupervisorAgent [58] have introduced real-time supervision to improve efficiency in multi-agent systems, though they do not utilize a contextual bandit formulation. To the best of our knowledge, we are the first to introduce the notion of adaptive supervision in the software engineering domain, proposing a contextual bandit-driven supervisor dedicated to maximizing the efficiency of automated test generation.

6 Conclusion↩︎

We introduced Scate, a supervisor-orchestrated agentic framework for automated unit test generation. Scate addresses the lazy generation phenomenon in modern coding agents, where agents prematurely terminate instead of generating tests for complex program logic. By formulating supervision as a contextual bandit problem, Scate replaces human intervention with an adaptive, metric-driven policy that continuously evaluates runtime coverage and structural testability to decide whether to continue standard generation, invoke program analysis, or terminate the generation process. Through this adaptive supervision, Scate balances code coverage and computational cost. Our evaluation on a diverse dataset constructed from Defects4J demonstrates that Scate significantly outperforms both standalone coding agents and state-of-the-art non-agentic approaches, improving line and branch coverage by 32.3% and 30.9%, respectively, over the standalone agent baseline. These results demonstrate that adaptive supervision is a key ingredient for realizing the full potential of agentic software testing.

References↩︎